Integrated system for quickly and accurately imaging and modeling three-dimensional objects

ABSTRACT

An integrated system generates a model of a three-dimensional object. A scanning laser device scans the three-dimensional object and generates a point cloud. The points of the point cloud each indicate a location of a corresponding point on a surface of the object. A first model is generated, responsive to the point cloud, that generates a first model representing constituent geometric shapes of the object. A data file is generated, responsive to the first model, that can be inputted to a computer-aided design system.

This application is a division of U.S. application Ser. No. 09/177,777,filed Oct. 23, 1998, now U.S. Pat. No. 6,473,079 which is a division ofPCT/US97/06793, filed Apr. 24, 1997, which is a continuation-in-part of08/638,961, filed Apr. 24, 1996, now U.S. Pat. No. 5,988,862.

TECHNICAL FIELD

The present invention relates generally to systems that document thegeometry and other attributes of objects in three dimensions and,specifically, to a system that employs a scanning lidar (range findinglaser) to quickly and accurately sense the position in three-dimensionalspace of selected points on the surface of an object to generate a pointcloud which represents the sensed positions of the selected points; thatrecognizes geometric shapes represented by groups of points in the pointcloud, and that generates a model that represents these geometricshapes. The model may be transformed into a further model usable bycomputer-aided design (CAD) tools, including conventional CAD tools.

BACKGROUND

Mapping the geometry (shape, dimensions and location) and otherattributes (e.g., color, texture and reflectance intensity) of complexreal objects (whether small components such as small mechanical parts orlarge objects such as buildings and sites) has conventionally been atedious and time consuming process. That is, such measurement havetraditionally been performed manually. In addition, transforming thesemeasurements into drawings or computer models required manual draftingor input into a CAD system for the production of the drawing or computermodels.

Recently innovations have endeavored to simplify this process, but allhave fallen short of achieving full integration, automation, precision,speed and range. For example, in the building industry, mapping astructure conventionally requires three basic steps:

1. Field data gathering

2. Data reduction and preparation

3. Drafting and CAD The field data gathering step is performed by a teamof surveyors who manually measure and record dimensions of pertinentcomponents of the structure such as walls, ceilings, beams, columns,doors, windows, fixtures, pipes, conduits and equipment. The surveyorsattempt to determine the geometry of the components as well as therelative location of the components in the structure. The surveyorsrecorded the data in a field notebook. The field-collected data is thenorganized and reduced to tables and organized sketches, and a CADoperator or drafter utilizes these tables to generate final drawings ormodels.

This process is labor intensive, time consuming, and error prone. Inaddition, using traditional surveying methods, the number of pointswhich can actually be measured is very limited, due to the high cost ofacquiring each point in terms of time and effort. Furthermore, if it isdesired to acquire color, texture and other attribute information,additional field notes must be taken (e.g., still photographs andvideo).

Recently, the field step has been somewhat automated by using a laserranging device built into or mounted on an electronic theodolite.Precision reflection targets (retro reflectors) are placed at thelocations of the object for which measurements are desired. Then, thelaser ranging device obtains a precise measurement of the distancebetween the instrument and the target, which the theodolite provides anaccurate indication of the horizontal and vertical angle offsets to thepoint relative to a given coordinate system. The distance and angle dataare either recorded automatically on a magnetic device connected to theinstrument or are reduced within the instrument to Cartesian coordinatesrelative to the instrument axes. This procedure is then repeated as manytimes as necessary to map a desired number of points of the object. Thecollected coordinates data can then be plotted directly on a CAD system.

Unfortunately, the plot is of little practical use since it does notindicate the object geometry. Moreover, because of the requirement forretro reflectors which must be manually placed, and because of therelatively long time per reading required by the laser range finder, thegathering of sufficient points to describe most objects is very laborintensive, time consuming and error prone.

Another known field gathering data process employs stereo photographyand aerial photogrammetry. That is, stereoscopic images are taken of theobjects and the resulting stereo photographs are registered eithermanually or using computerized techniques to reproduce the relativelocation of the camera picture plane location at the time eachphotograph was taken. The data reduction and preparation step isperformed manually by a specially trained operator. Specifically, withthe aid of specially mounted stereoscopic viewing lenses, the operatordigitizes the coordinates of a sufficient number of points to allow thedefinition of the objects using the stereo photographs. Again, thedigitized data is input into a CAD system or is manually drawn on paper.

SUMMARY

The present invention is an integrated system for generating a model ofa three-dimensional object. A scanning laser device scans thethree-dimensional object and generates a point cloud. The points of thepoint cloud each indicate a location of a corresponding point on asurface of the object. A first model is generated, responsive to thepoint cloud, representing constituent geometric shapes of the object. Adata file is generated, responsive to the first model, that can beinputted to a computer-aided design system.

The subject invention further includes a method of controlling thetiming of output pulses from a laser for use in a device which requiresscanning of the laser output, wherein each output pulse is generated inresponse to a pump pulse comprising the steps of: monitoring the timedelay between the initiation of the pump pulses and the subsequentgeneration of the associated output pulses; predicting the time delaybetween the initiation of next pump pulse and the associated outputpulse based on the monitored time delays and; initiating the next pumppulse at a time selected to insure the output pulse is-generated at atime to permit proper positioning of the laser output during the scan ofthe beam.

The present invention further includes a method of manually separatingfrom a plurality of clouds of points, representing three-dimensionalfeatures in a scene, a subset of the points that represents a desiredfeature in the scene, the method comprising: selecting all the pointclouds that include at least some data points representing the desiredfeature; and changing a view of the clouds and drawing a polygonal lassoto refine a selected subset of points to be included in a pointsub-cloud and repeating the refining as many times as required to obtainthe desired sub-cloud.

The present invention further includes a method for automaticallysegmenting a scan field of a scene into subsets of points that representdifferent surfaces in the scene, comprising the steps of: separating thescan field into a depth grid that includes depth information for scannedpoints of surfaces in the scene and a normal grid that includes anestimate of a normal to scanned points of the surfaces; convolving thedepth information of the depth grid to generate a depth rating imagewhose values represent a gradient of depth change from one scanned pointto another scanned point in the scene; convolving the components of thenormal grid to generate a scalar value for each component for each pointof the normal grid; for each point of the normal grid, determining fromthe scalar values for the components of that particular point a gradientof the normal at that point, wherein the gradients determined for thepoints of the normal grid collectively constitute a normal rating image;converting the depth rating image to a binary depth image using arecursive thresholding technique; converting the normal rating image toa binary normal image using a recursive thresholding technique;combining the binary depth image and the binary normal image todetermine a single edge image; and grouping subsets of non-edge pointsas belonging to corresponding surfaces of the scene.

The method can further include the steps of determining the type ofgeometric primitive that would best first each group of points; fittingthe geometric primitive to the data points; and intersecting adjacentplanar regions in the scene.

The subject matter further includes a method for fitting a point cloudrepresenting a corner, comprising: determining a fit of three planes tothe points of the point cloud and creating the planes for a model;determining the three lines at the intersection of pairs of planes andcreating the lines for the model; and determining the vertex point atthe intersection of the three planes and creating a vertex point for themodel.

The subject invention further includes a method for modeling athree-dimensional scene, comprising: generating a plurality of pointsthat each represent a point on a surface of the scene; determining abest fit of a cylinder for a group of the points using surface normalestimates and global error minimization.

The subject invention further includes a method for modeling athree-dimensional scene, comprising: generating a plurality of pointsthat each represent a point on a surface of the scene; determining abest fit of a cylinder for a group of the points using a quadric surfacefit and global error minimization.

The subject invention further includes a method for modeling athree-dimensional scene, comprising: generating a plurality of pointsthat each represent a point on a surface of the scene; determining abest fit of a sphere for a group of the points using a quadric surfacefit and global error minimization.

The subject invention further includes a method for modeling athree-dimensional scene, comprising: generating a plurality of pointsthat each represent a point on a surface of the scene; determining abest fit quadric surface for a group of points; and determining whichgeometric primitive of a plurality of the family described by thequadric surface best fits the group of points.

The subject invention further includes a method for merging twogeometric primitives of the same type to form a single geometricprimitive of the type, comprising: creating a new group of points bycombining the points used to originally fit each of the two primitives;and fitting the new geometric primitive using any appropriate fittingtechnique and the newly generated point group with points from each ofthe original primitives.

The subject invention further includes a method of registering a firstmodel, consisting of a plurality of points and geometric primitives andhaving a first coordinate system, with a second model, consisting of aplurality of points and geometric primitives and having a secondcoordinate system, comprising: identifying by a user common features ofthe first and second scenes; identifying a transformation betweencoordinate systems that is responsive to the identification; andtransforming the objects of the second model so that they use the firstcoordinate system.

The subject invention further includes a method of warping, comprising:selecting one or more models represented by a plurality of point cloudsand geometric primitives; specifying constraints on the locations of anynumber of points or geometric primitives; creating an artificial volumethat surrounds the points and geometric primitives in each view andassigning mechanical material characteristics to the surrounding volume;computing a minimum energy configuration for the material in thesurrounding volume in which the points or geometric primitives areembedded such that the configuration satisfies all applied constraints;and displacing the points and primitives in accordance with the computedminimum energy configuration of the surrounding volume of material. Inthe latter method, the constraints can be specified to eliminate closureerrors.

The subject invention further includes an integrated system forgenerating a model of a three-dimensional scene, comprising: a scanninglaser device that scans the three dimensional scene with pulsed laserbeam wherein the pulses of light last less than 1 nanosecond with up to0.2 _(μ)J in each pulse and measures the time delay, with a resolutionof 30 psec or less, between each emitted pulse and a corresponding pulsethat is returned from a surface of the scene and wherein said scanninglaser device further tracks and measures the angular orientation of thebeam during the scan; and means for generating a point cloud based uponthe measured time delays and angle measurements, the point cloudcomprising a plurality of data points that each represents a location ofa corresponding point on the surface.

The subject invention further includes a system for calibrating themeasuring electronics in a device which requires monitoring the time offlight of the output pulses from a laser comprising: a single modeoptical fiber with one end thereof positioned to receive the outputpulses of the laser, said single mode optical fiber having a knownlength; a detector positioned at one of the ends of the fiber formonitoring when the pulses exit the fiber and generating a signal inresponse thereto, said signal being passed through the measuringelectronics; and a processor for calculating a theoretical length of thefiber based on the detection of the pulse exiting the fiber andcomparing that calculated length with known length of the fiber tocalibrate the measuring electronics.

The optical fiber can include partial reflectors located at each endthereof so that for each laser pulse entering the fiber a train ofpulses will exit the fiber and wherein said train of pulses are used tofurther calibrate the measuring electronics.

The system can further include delay measurement electronics and whereinthe train of pulses have a fixed delay therebetween whereby themonitoring of the train of pulses can be used to calibrate the delayelectronics.

The system can further include a means for varying the power of thepulses monitored by the detector and wherein said detector functions togenerate a signal when the power of the detected light exceeds apredetermined threshold and wherein said processor functions to trackthe variation in the delay of the generation of the output signal by thedetector as a function of the power of the output pulses, said processorfurther functioning to calibrate the measurement of the delay based onthe measured power of successive pulses used for monitoring the time offlight.

The subject invention further includes an apparatus for obtainingposition information about surface points of a three dimensional objectcomprising: a laser for generating an output beam; scanning system formoving the laser beam over the object; monitoring system forautomatically measuring the range to the object based on the measurementof the reflection of the laser beam, said monitor system also trackingand measuring the angular position of the laser beam, said monitoringsystem having a positional accuracy for each point in three dimensionalspace equal to or better than six millimeters at one standard deviationover a range of up to 100 meters.

Each range measurement can be made in under 0.005 seconds. The laser cangenerate a pulsed output and the energy per pulse can be less than 0.2micro joules and the average output power of the laser can be less than1.0 milliwatts.

The subject invention further includes an apparatus for measuring thedistance to an object comprising: a laser for generating a beam ofoutput pulses; a monitoring system for measuring the distance to theobject based on the reflection of the laser beam, said monitoring systemhaving an accuracy equal to or better than 6 millimeters at one standarddeviation over its entire range of up to 100 meters and wherein eachmeasurement can be made in less than 0.005 seconds and wherein the laserhas an energy per pulse of no more than 0.2 micro joules and an averagepower of no more than 1 milliwatt. If the object is provided with retroreflectors and where the range of operation is up to one mile.

The subject invention further includes an apparatus for acquiring threedimensional information from a remote object comprising: a scanninglaser module for measuring position information of the object; a videomodule for capturing image information from the object; and a processorfor rendering a model of the object which includes the positioninformation and the image information.

The video image information can be collected in a spatially coincidentmanner with the measurement of position information. The video imageinformation can be collected from points adjacent to the points whereposition information is obtained.

The subject invention further includes an apparatus for obtainingpositional information about surface points of a three dimensionalobject comprising: a scanning module for measuring three dimensionalposition information about an object; a video module for capturing anddisplaying image information from the object; and a processor operatingwith the scanning and video modules and permitting the use of the imageinformation captured by said video module to aid in targeting thescanning module. The processor can function to specify a portion of theobject to be targeted by the scanning module by dragging the image of anoutline over the video image of the area to be targeted.

The subject invention further includes an apparatus for obtainingpositional information about surface points of a three dimensionalobject comprising: a scanning module for measuring three dimensionalposition information about an object; a video module for displayingimage information obtained from the scanning module; a processoroperating with the scanning and video modules and permitting the use ofthe image information displayed by said video module to further refinethe targeting of the scanning module.

The subject invention further includes an apparatus for obtainingpositional information about surface points of a three dimensionalobject comprising: a scanning module for measuring three dimensionalposition information about an object, said scanning module including alaser for emitting a beam of visible radiation; and a processor forcontrolling the scanning module and wherein said laser can be manuallypositioned so that the visible beam will target the portion of theobject to be scanned in response to a control signal from the processor.

The subject invention further includes a system for calibrating themeasuring electronics in a device which requires monitoring frequencychanges in a light beam generated by a laser used to measure distance toan object, wherein said beam has frequency chirp imposed thereoncomprising a single mode optical fiber with one end thereof positionedto receive light from the laser; a detector positioned to receive lightwhich has traveled through and exited the fiber in combination whichlight from the laser which has not traveled through the fiber, saiddetector for monitoring the changes in the frequency of the combinedbeam; and processor for determining the linearity of the chirp on thebeam based on uniformity of the frequency changes measured by thedetector and using the result to calibrate the measuring electronics.

The fiber can have a known length and includes a partial reflector onsaid one end and at least a partial reflector on the other end, andwherein light reflected from said one end of the fiber which has nottraveled in the fiber is measured by the detector and wherein theprocessor further functions to calculate a theoretical length of thefiber based on the frequency changes measured by the detector andcompares that calculated length with the known length of the fiber tocalibrate the measuring electronics.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a system in accordance with an embodimentof the invention.

FIG. 1A shows the overall flow of how one may use an embodiment of theinvention to scan an object, organize acquired points, fit geometricshapes to the organized point, manipulate the fitted geometric shapes,and display the resulting manipulated geometric shapes.

FIG. 2 is a more detailed block diagram of the system of FIG. 1.

FIGS. 3A and 3B show the physical arrangement of the FDV of the FIG. 1system, and also shows how the FDV is coupled to the tripod by a forkmount.

FIGS. 4A and 4B show an example coordinate system relative to the FDV ofthe FIG. 1 system.

FIG. 5 is a block diagram of one embodiment of an FDV in accordance withthe invention.

FIG. 6 is a block diagram of the optical transceiver of the FIG. 5 FDV.

FIG. 6A shows a dual mirror arrangement of the scanner shown in FIG. 6.

FIG. 7 is a block diagram which shows an embodiment of the laser.

FIG. 7A is a block diagram of an embodiment of the beam expander shownin FIG. 6.

FIG. 8 shows an embodiment of the duplexer.

FIG. 8A shows a partially-reflecting duplexer.

FIG. 9 shows an embodiment of the window of the FIG. 8 duplexer.

FIG. 10 is a flowchart that shows calculations performed by the FDV DSP.

FIGS. 11A and 11B show a unidirectional scan pattern and abi-directional scan pattern, respectively.

FIG. 12 is a block diagram of an embodiment of the FDV processor.

FIG. 13 is a block diagram of example circuitry for determining adesired position of an FDV mirror.

FIG. 14 is a block diagram of an example signal conditioning and energyintegration circuit of the timing circuit shown in FIG. 12.

FIG. 15 is a detailed block diagram of the system of FIG. 1.

FIGS. 16A and 16B show two windows used to operate the CGP.

FIGS. 17A and 17B show a targeting box and a point cloud.

FIG. 18, shows a point cloud from the surface of a horse sculpture.

FIG. 19 shows the point cloud of FIG. 18 color mapped with the laserreturn intensities.

FIG. 20 shows a cloud of points from a corner feature.

FIG. 21 shows the cloud of points of FIG. 20 and a polygonal lasso usedfor manual segmentation.

FIG. 22 shows the cloud of points of FIG. 20 segmented into foursubgroups, three subgroups on the surfaces of planes and a subgroup ofedge points that are not part of the plane.

FIG. 23 shows the cloud of points of FIG. 20 rendered as a triangulatedmesh.

FIG. 24 shows the corner feature of FIG. 20 with planes fit to thegroups of cloud points.

FIG. 25 shows a point cloud from the surface of a cylinder.

FIG. 26 shows a cylinder primitive that was fit to the points shown inFIG. 25.

FIG. 27 shows a cloud of points from the surfaces on a piping system.

FIG. 28 shows cylinder primitives that were fit to the points shown inFIG. 27.

FIG. 29 shows the completed piping model, after extending pipes andadding elbows.

FIG. 30 shows the result of a corner fit, giving three planes, threelines, and a vertex.

FIG. 31 shows a cylinder primitive in a scene.

FIG. 32 shows the cylinder form FIG. 31 extended to meet adjacentobjects.

FIG. 33 shows a cloud of points from the surface from a variety ofobjects.

FIG. 34 shows a model containing primitives that were fit to the pointsshown in FIG. 33.

FIG. 35 shows configuration of a frequency adjustable laser.

FIG. 36 shows block diagram of conventional FM chirp lidar.

FIG. 37 shows block diagram of self-calibrating FM chirp lidar.

FIG. 38 illustrates the relative timing at which a large and a smallpulse cross a predetermined threshold.

FIG. 39 illustrates one circuit for measuring pulse energy.

FIG. 40 illustrates another circuit for measuring pulse energy.

DETAILED DESCRIPTION

A. Overview

1. Overall System

FIG. 1 is a block diagram that illustrates the invention in its broadestaspect. Referring to FIG. 1, a Field Digital Vision (FDV) module 10includes a scanning sensor for scanning an object 20 and for sensing theposition in three-dimensional space of selected points on the surface ofthe object 20. The FDV module 10 generates a point cloud 30 whichrepresents the sensed positions of the selected points. The point cloud30 also represents other attributes of the sensed positions, such asreflectivity, surface color and texture.

A Computer Graphics Perception (CGP) module 40 interacts with the FDV toprovide control and targeting functions for the FDV module 10 sensor. Inaddition, using the point cloud, the CGP module 40 recognizes geometricshapes represented by groups of points in the point cloud 30, and theCGP module generates a CGP model 42 that represents these geometricshapes. From the CGP model 42, the CGP module 40 generates a furthermodel usable by computer-aided design (CAD) tools 50. The CAD tools maybe conventional.

FIG. 1A shows the overall flow of how one may use an embodiment of theinvention to scan an object, organize acquired points, fit geometricshapes to the organized point, manipulate the fitted geometric shapes,and display the resulting manipulated geometric shapes.

2. FDV Module Overview

Referring to FIG. 2, the FDV 10 includes a scanning laser system (lidar)210 that scans points of the object 20 and that generates a lidar datasignal that precisely represents the position in three-dimensional spaceof each scanned point. The lidar data signal for groups of scannedpoints collectively constitute the point cloud 30. In addition, a videosystem 220, preferably including both wide angle and narrow angle CCDcameras, is provided. The wide angle CCD camera of the video system 220acquires a video image of the object 20 and provides, to the CGP 40 viaa control/interface module 230 of the FDV 10, a signal that representsthe acquired video image.

In response to user input relative to the signal that represents theacquired video image, the CGP 40 provides a scanning control signal tothe lidar 210, via the control/interface module 230, for controllingwhich points on the surface of the object 20 the lidar 210 scans. Moreparticularly, the scanning control signal provided from the CGP 40controls an accurate and repeatable beam steering mechanism to steer alaser beam of the lidar 210.

In addition, the narrow angle CCD camera of the video system 220captures the texture and color information, and provides this capturedinformation to the CGP 40.

3. CGP Module Overview

Referring still to FIG. 2, the CGP 40 is constituted of a dataprocessing system (e.g., a notebook computer or a graphics workstation)and special purpose software that when executed conFig.s the CGP 40 dataprocessing system to perform the FDV 10 control and targeting functions,and also to perform the CGP model 42 generation functions.

i. FDV Control

The CGP 40 controls the scanning lidar 210 of the FDV 10 by providing alidar control signal to the FDV 10 that controls which points of theobject 20 the FDV 10 scans. User input is provided to the CGP 40, whichdefines what portions of the object 20 to scan, and at what resolution.

ii. Model Generation

Each data point in the point cloud 30 generated by the FDV 10 representsboth distance to a corresponding laser impingement point from an FDV 10“origin point” and the angle from the origin point to the laserimpingement point. The CGP software conFig.s the CGP 40 computer toprocess the data points of the point cloud 30, generated by the lidar210 as a result of scanning the object 20, to display and visualize thescanned portions of the object 20. More specifically, the CGP softwareconFig.s the CGP 40 computer to recognize geometric shapes in the object20 (“graphic perception”) and, using these recognized geometric shapes,to perform geometry construction, 3D model construction, 3Dvisualization, and database functions for automated acquisition ormanual input of object attributes, generation of plans, sections, anddimensions, data query, and CAD interfaces, and networking options.

B. Details

1. FDV Module Detail

FIG. 5 is a block diagram of one embodiment of an FDV 10 in accordancewith the invention. A lidar transceiver 502 includes a laser, transmitoptics, receive optics and detector for generating ranging and intensitydata. A scanning system 504 includes dual orthogonal scanning mirrors,galvo motors, and encoders for steering the laser beam and determiningthe azimuth and altitude angles of the laser beam from the positions ofthe mirrors. A wide angle video system 506 generates targeting videoinformation and a narrow angle video system 507 generates color andtexture information. Control/interface circuitry 230 handles theexchange of data between the FDV 10 and the CGP 40.

If the laser beam is quasi-CW, always on with either intensitymodulation (AM) or wavelength modulation (FM), the distance to theobject 20 can be inferred by any of a number of techniques involvingdemodulation at the transceiver 502. If the laser is pulsed, thedistance to the object 20 is usually measured by the time of flight fromthe transceiver 502 to the object 20 and back. Other laser modulationschemes may be used.

In the time-of-flight embodiment, the laser is preferably of the typedisclosed in U.S. Pat. Nos. 5,132,977; 5,386,427; and 5,381,431,assigned to Massachusetts Institute of Technology. In particular, thebeam generated by such a laser has special properties such as beingcapable of producing pulse widths less than 1 nsec.

A particular embodiment of the laser which has been used is particularlysuitable for precision lidar since:

1. The short pulsewidth provides high accuracy, since radar theory showsthat the accuracy is proportional to the inverse of the pulse time.

2. The laser itself is physically quite small, especially useful forportable applications.

3. It has a diffraction limited beam, which implies that the spot sizeat a distance is not limited by the properties of the light, but only bythe quality of the optics used to collimate or focus it.

4. Since the wavelength is quite short (532 nm), and Rayleigh range isinversely proportional to wavelength, the beam can be kept small over alarge distance interval. In fact with a 1 cm exit aperture, the beamwill remain less than 6 mm over 50 m.

In one preferred embodiment, the laser beam is directed by theorthogonal scanner mirrors to a laser impingement point on the surfaceof the object 20. The range can be determined by any of a number ofconventional lidar techniques. For example, the “time of flight” of alaser pulse to travel from the laser to the surface of the object 20,and then back to the detector, is determined. The range is determinedbased on the constant speed of light, with appropriate adjustments beingmade for atmospheric factors.

A system in accordance with the present invention can provide highranging accuracy at high acquisition rates. For example, at 100 mranges, a 1 mm accuracy can be achieved on a single shot basis, withanywhere from 1000 to 5000 data points being acquired per second.

In other embodiments, a chirp lidar may be employed. The essentialcomponent of a chirp lidar can be modulated with a linear change ofwavelength over time. Thus, the wavelength of the light emitting fromthe laser will be given by λ(t)=k(t-t₀)+λ₀. In practice, such a laser iscommonly manufactured by making a composite of two materials, a typicallaser gain media such as NdYAG (3510), and substance which can have itsindex of refraction altered electrically such as Lithium Niobate (3520).(See FIG. 35) This effectively changes the length of the laser cavity,and hence alters the emitted wavelength. Commercially available laserscan be modulated up to about 100 GHz with a voltage modulation of about1 kV, so the frequency of the light will be approximatelyf(t)=k(t-t₀)+f₀.

Referring to FIG. 36, in a typical FM chirp system, a portion of thelight emitted by the laser 3610 is sampled and recombined at the beamsplitter 3630 with the light returning from the target 3620. Since thelight is delayed by the amount of time to contact the target and return,the light returning from the target will have a lower frequency than thelight which is sampled from the laser. This difference will be apparentin the output of detector 3610 which measures the intensity of thecombined beams. If the frequency ramp of light is exactly linear and thelaser has a coherence length much greater than the distance to thetarget, combining the beams will produce a constant measured frequencyfrom the detector which is proportional to the range: f=k₁* d+d₀.Chirped Yag lasers as described above have a coherence length of about20 km, but the chirp is not linear, and this has severely limited theaccuracy of existing FM chirp lidars.

Referring to FIG. 37, a huge improvement in accuracy can be realized byadding a system to calibrate every range measurement. A fiber isprepared which has a partial reflector 3771 on one end, and a nearlytotal reflector 3772 on the other. Now a portion of the light emitted bythe laser 3710 is sampled and recombined at beam splitter 3740 with thelight returning from the target 3720, and the intensity is measured by adetector 3760. An additional sample of the beam emitted from the laseris sampled by beam splitter 3730 and introduced into the fiber at thepartially reflected end 3771. The beam propagates a fixed distance downthe fiber and reflects off the end face and is recombined with the beamwhich is reflecting off the partially reflecting face 3771, and ismeasured with a second detector 3750. The linearity of the chirp is thenmeasured by determining the deviation from a constant frequency of theoutput of detector 3750, and this information is used to correct for theeffects of the nonlinear chirp in the output of detector 3760, whichcorresponds to the target range measurement.

Referring to FIGS. 3A and 3B, in the embodiment, the FDV 10 isphysically housed in a box 330 made of metal or other suitable housingmaterial. The box 330 is suspended from its side panels by a fork mountmechanism 310. The fork mount system is supported on a turntable 340,and the turntable 340 is mountable on a tripod 320. Using the forkmechanism 310, the FDV 10 can be rotated horizontally (“azimuthrotation”) and vertically (“elevation tilt”, or “altitude”). Generally,a position of the tripod 320 is referred to as a “setting” or“positioning”; the rotation and tilt of the FDV 10 in the fork mount 310is referred to as “pointing” or “orientation”. A “view” is generallyassociated with a given setting and orientation.

The fork mount system 310 preferably includes high accuracy azimuth andaltitude rotation measuring devices (e.g., conventional theodolite-typeoptical or electronic encoders) to provide precise rotation and tiltdata of the FDV 10. This feature can allow the automatic integration ofscans taken from the same tripod 320 setting, but with a differentorientation of the FDV 10. In the event these devices are not used, andfor scans taken from different settings and orientations, these scanscan be integrated using techniques described later in this disclosure.

It should be noted at this point that, while conventional surveyinginstruments should be leveled prior to operation in order to operateproperly, this is not a requirement of the FDV 10. This is due to thenovel methods of this invention as embodied since its own internalcoordinate system and the procedures utilized in its software that takeadvantage of its method of acquiring position data. The system, however,does have the ability to be leveled and to be used in a manner similarto a traditional theodolite. The Cartesian coordinates of the FDV 10 inthe example embodiment are shown in FIGS. 4A and 4B.

Referring still to FIGS. 3A and 3B, in one embodiment, two orthogonalmirrors of the FDV 10 provide a field of view of approximately 40° by40° (“Field of View”, or “View” is defined as the maximum size of thearea projected by the laser maximum deflections of the beam in degrees).The field of view can be increased or decreased by resizing the mirrorsand certain parts of the optical train. The fork mount described aboveis utilized to allow pointing of the FDVs 40°×40° field of view anywhereover a projected sphere thus affording a wide range of flexibility inimaging large objects or groups of objects from the same setting. Othermounting methods may be used to accomplish the same purpose.

High accuracy and repeatability electronic encoders read the rotationalangles of the orthogonal mirrors, and the readings of the mirrorrotation angles are precisely timed to coincide with the range reading.Preferably, the system is Class II FDA eye safe. A first embodiment has_(±)6 mm spatial accuracy over a range of _(±)50 m. In anotherembodiment, autofocus capability and 5-6 picosecond electronics areincluded, which extends the system's range accuracy to _(±)1 mm and_(±)1 mm spatial accuracy over _(±)50 m. The range (and accuracy) of thesystem can be significantly influenced by the choice of eye safetyclassification selected, but these limitations are not inherentlimitations of the invention itself.

The following is a description of the key components of preferredembodiment of the FDV 10. A block diagram of the optical transceiver 502of the FDV 10 is shown in FIG. 6. The optical transceiver 502 transmitsan optical pulse to a spot on object 20, and receives a reflectedoptical pulse from the object 20. Given the constant speed of light, theoptical transceiver calibrates the distance to the spot on the target.

Referring to FIG. 6, the laser 602 fires an optical pulse which lastsless than 250 psec, in response to an external command provided from alaser controller 604. The laser 602 produces a pulse, preferably at awavelength of 532 nm, within 100-300 μsec after an external signalemanating from a digital signal processor which provides central controlof real time events. The time delay is a complicated function of recentlaser history and environmental conditions. This function is notcompletely known at present. However, a software algorithm, which isdescribed elsewhere, is employed to estimate the time delay withadequate accuracy for the required measurements.

The laser beam output of the laser 602 is transmitted through a beamexpander 606 that is focused to adjust the size of a light spot thatwill eventually impinge upon a point on the object 20. The focussedoptical pulse is then transmitted through a duplexer 608, which is anoptical system for aligning the outgoing optical path with the incomingoptical path. The duplexer 608 directs a significant first portion ofthe light energy of the outgoing optical pulse to a spot on the object20 via a scanner 614, but a second, much smaller portion, of the lightenergy of the outgoing optical pulse is directed to a receiver telescope610. The portion of the outgoing optical pulse that propagates to theobject 20 impinges on the spot on the object 20, and some of the energyof the optical pulse is reflected off the object 20 in a direction backto the duplexer 608. The returning optical pulse is directed by theduplexer 608 to a receiver telescope 610, which focuses the receivedenergy onto a detector 612. The detector 612 converts the receivedoptical pulse energy into electrical energy, and the output of thedetector 612 is a series of electrical pulses, the first (which isgenerated by the detector 612 in response to the second, small portion,of the transmitted pulse not directed toward the object 20) occurs at ashort fixed time (i.e., fixed by the length of the optical path throughthe beam expander 606, the duplexer 608 and the receiver telescope 610)and the second of which occurs as light energy returns from the object20. Both the second, small portion of the transmitted pulse not directedtoward the object 20, and the return optical pulse reflected from thespot on the object 20, are provided to the timing circuit 616 whichcalculates the time of flight to the spot on the object 20. The range tothe spot on the object 20 can then be readily calculated from thecalculated time of flight.

FIG. 7 is a block diagram which shows an embodiment of the laser 602.The heart of the laser system 702 is a conventional laser chip 702 thatinclude two bonded crystals coated with antireflective dielectriccoatings on their faces. The laser chip 602 is pumped with a solid statediode 704 operating at 808.5 nm_(±)3 nm. The output frequency of thediode pump 704 is adjusted by changing its temperature with athermoelectric cooler 706. The temperature of the diode pump 704 ismeasured with a thermistor 708, and the measured temperature is fed backinto the diode power supply 710. The required temperature varies witheach individual diode, but it typically ranges from 20° to 30° C.

The output power of the diode pump 704 is typically 1 watt, launchedinto a 100 μm core glass fiber. When continuously pumped, the output ofthe crystal laser 602 is approximately 35 mW average power at 1.064 μm,which corresponds to 2.4 μJ pulses lasting about 280 psec at arepetition rate of 15 kHz. The multimode fiber is preferably terminatedby an SMA905 solid brass connector, with the crystal of the laser chip702 glued to one end of the connector with an optical resin. Thusensures adequate thermal dissipation from the crystal of the laser chip702, keeping the crystal 702 within the temperature range required formost efficient operation.

A piece of KTP frequency doubling crystal 712 is held within a fewmillimeters of the face of the laser chip crystal 702. This provides anultimate output from the laser 602 having a 12 mW average power at 532nm, which corresponds to 0.8 μJ pulses lasting approximately 218 psec.This ultimate output from the laser 602 is nearly diffraction limited(i.e., one which has theoretically minimum divergence, given a specificwavelength and waist diameter), with an apparent waist diameter of 56μm.

Embodiments of the invention which meet FDA Class II eye safe systemdesign specifications are potentially more commercially viable. In orderto meet this specification, the maximum energy per pulse that can betransmitted at 532 nm is 0.2 μJ. With this restriction, the averagepower transmitted is largely dependent on the pulse repetition rate, andis given by the following table

ENERGY PER AVERAGE REPETITION CLASS PULSE POWER RATE I .2 μJ .39 μW 1.95Hz IIA .2 μJ 3.9 μW 19.5 Hz II .2 μJ 1.0 mW   5 kHz IIIA .2 μJ 5.0 mW  25 kHz

In one embodiment of the invention, the beam expander 606 is entirelyconventional (e.g., Melles Griot model number 09LBM013, 10× beamexpander). The transceiver 502 has cross axis accuracy which isproportional to the size of the laser beam impinging on the intendedtarget The base design of 6 mm accuracy has a simple beam expander. Thelaser 602 can be collimated with a fixed 10× beam expander 606 which hasan aperture of <1 cm to produce a beam whose 1/e^(2*power) beam width isless than 6 mm over a range of 50 m.

FIG. 7A shows a further embodiment 750 of the beam expander 606 thatincludes features which allow the system of the invention to measureranges at an accuracy of approximately 1 mm at 50 m. This is because theimpingement spot on the object 20 of the laser beam expanded by aconventional beam expander is collimated, and produces a spot of no morethan 6 mm over a 50 m range. However, a beam can be focused through a 50mm aperture to a spot of size no more than 1 mm over a 50 m range—butthe spot will be much larger at other ranges. Thus the beam expander 750of a system having 1 mm accuracy at 50 m includes a movable opticalelement 752 which can change the size of the focused spot. Additionally,the beam expander 750 includes an adjustable aperture 755, and means forcontrolling the adjustment, so that the distance, from the laser, overwhich the beam stays at 1 mm in diameter remains approximately constant.The minimum diameter spot produced with a diffraction limited lens offocal length f and diameter D is d₀=2fλ/D. The Rayleigh range of thefocused spot, which is the depth of focus of the beam, is given byb=2π_(ω0) ²/λ=2πf²λ/D². Thus, if f/D is held constant, the depth offocus will not be a function of the range of the focused spot, f.

As the beam focus is changed, the elements should stay sufficientlyaligned so as to prevent the beam from changing direction by more than afraction of 1 mm at 50 m, or this will appear as an error in theplacement of the point in space. In order to minimize this beam wander,a linear servo motor 754 (see FIG. 7A) is employed for controlling theposition of the focusing mechanism, and a transducer provides positionfeedback. The lens 752 is mounted in an annular ring 753, which preventsit from rotating or misaligning while it is being translated.

Duplexer

An embodiment of the duplexer 608 is shown in FIG. 8. The optical systemof the duplexer 608 is conFig.d such that the outgoing beam from thebeam expander 606 to the scanner 504 is coaxial with the return beamreflected from the object 20. Thus, only one scanner 504 need beprovided. In the embodiment of the duplexer shown in FIG. 8, a window802 is provided, with a 50% beamsplitter 804 attached over the window802. When an optical pulse is transmitted from the laser 602 and throughthe beam expander 606, the pulse impinges upon the beam splitter 804.Most of the light energy of the pulse is reflected off the beam splitter804 and is passed to the scanner 504, but some of the optical pulseproceeds through the beam splitter 804 and impinges upon a lowreflectance beam block 806. Due to the low (but non-zero) reflectance ofthe beam block 806, a small fraction of the optical pulse hitting thebeam block 806 returns to the beam splitter 804 and is reflected intothe receiver 610.

Moreover, as an optical pulse returns from the object 20, since only thecentral portion of the return pulse is obscured by the prism 804, mostof the light impinging on the window 802 makes its way to the receiver610.

Partially Reflecting Duplexer

Referring now to FIG. 8A, for the 1 mm accuracy embodiment, apartially-reflecting duplexer 850 is employed. With this duplexer, afraction of the light pulse provided from the beam expander into a beamstop 652 and reflects off the duplexer window 850 to the receivertelescope 610. The remainder of the light pulse proceeds to the object20. Most of the return light pulse from the object 20 continues onthrough the window 850 and is collected by the receiver telescope 610.The window 850 is AR coated on the receiver side, and partially mirroredon the laser side. The entire window 850 is used to steer the outgoingbeam, since a 50 mm aperture is required to focus the spot to 1 mm at 50m. The partial reflectance is chosen in view of the laser transmissionpower and the applicable eye-safe classification level. For example, ifthe laser transmission power is four times the allowable level of theapplicable eye-safe level, then the partial mirroring is chosen toreflect 25% and absorb 75%.

Referring now to FIG. 9, in the 6 mm embodiment, improved efficiency canbe achieved in collecting the return optical pulse if only the center ofthe window 802 is coated 904 to reflect the outgoing pulse, and theremainder of the window 802 is anti-reflective coated 906. In this way,the return optical pulse is not reflected out of the receiver by thepart of the window 802 that is antireflection coated 906.

Preferably, the laser 602 emits a strongly polarized beam so that thereflective coating 904 can be optimized to have slightly differentreflection coefficients for the two planar polarizations (20%-S and30%-P). In such an embodiment, the power of the beam impinged onto theobject 20 can be fine tuned merely by physically rotating the laserbody.

Receiver Telescope

Referring again to FIG. 6, after the returning pulse has passed throughthe duplexer 608, it is collected by the receiver telescope 610, whichoptimizes the amount of signal provided to the detector 612. Thereceiver telescope 610 may be a simple 50 mm aperture lens. The lens ispreferably selected so that the variation in pulse energy entering thedetector 612 does not change as a function of the distance to the object20 over the range of distances for which the instrument is designed. Amultiple element lens can be designed to minimize the variation inreceived pulse energy as a function of range somewhat more effectivelythan a single element lens. That is, at the greatest expected distance,the focal length of the lens is such that all the incoming light, whichis effectively collimated since it is generated by a point source in thefar field, is focused to completely fill the detector 612. As the object20 becomes closer to the telescope 610, the spot of return light becomeslarger than the detector 612. The power incident on the detector 612increases as the square of the distance from the telescope 610 to theobject 20, up to the maximum expected distance. Moreover, the powerreturning from the object 20 decreases as the square of the distancefrom the telescope 610 to the object 20. Thus, in practice, these twoeffects approximately cancel each other. This minimizes the variation inoptical power incident on the detector 612 over the range of anticipateduse. In the 11 mm option, the receiver optics can be improved in somecases by using a two element, adjustable focus, Newtonian telescope(e.g., similar to the 1 mm beam expander).

Detector

The detector 612 converts optical pulses to electrical pulses which canbe processed by the elapsed time measurement electronics (timing circuit616). In one embodiment, the detector 612 is an avalanche photodiode(APD) with greater than 1 GHz electrical bandwidth. In addition to thetime between the start and any stop pulses, the intensities of all thepulses are recorded. The intensity information is used to make acorrection to the range measurement derived from the timing information.

Scanner

The scanner 504 may be conventional. The scanner 504 directs theoutgoing pulses from the duplexer 608 to a desired position on theobject 20 and directs the incoming return pulse into the receivertelescope 610. The scanner 504 directs light to the narrow field ccdcamera 507 to collect color and texture in the immediate vicinity of thescanned laser points, which provides for precise registration of colorand texture obtained with the lidar acquired point geometry. In oneembodiment, the scanner 504 includes a dual mirror arrangement (see FIG.6A) for beam steering, although any conventional high accuracy andrepeatability beam steering mechanism may be employed. The dual mirrorarrangement includes two mirrors which are rotated on orthogonal axes bymoving coil motors. These motors have an integral position decoder whichhas angular repeatability of less than 1 microradian. The mount for thescanners is integrally formed with the supports for the laser and otheroptics. This system provides 40 degrees of optical motion in bothaltitude (elevation) and azimuth at several Hertz.

Electronics

A. Timing Circuit

Another embodiment of the scanner 504 mechanism consists of a singlemirror rotating about a central axis, mounted on a rotating turret Inthis configuration, the physical coordinate system would be spherical,with the faster (due to the less inertia) mirror providing theelevational angle and the more slowly rotating turret providingazimuthal motion. A system such as this could provide a field of view ofmore than 90 degrees in a vertical plane and a full 360 degrees in ahorizontal plane (both planes being relative to some chosen scannercoordinate system.

Electronics

Ranging Electronics

The function of the ranging electronics is to compute the range from theFDV 10 to the object 20 based upon the electrical output of the detector612. Several possible methods may be used, including a demodulator inthe case of a quasi-CW modulated laser system. For the preferred time offlight embodiment, an interval timer (timing circuit) measures therelative time interval between an initial (start) pulse reflecteddirectly into the receiver 610 by the duplexor 608, and the pulsereflected off of the object 20 back into the receiver 610.

Reflectivity Electronics

In many cases, it is useful to know not only the position in space of apoint on the object 20, but also know the reflectivity (at someparticular wavelength) of that point. The reflectivity electronicsmeasure the amount of light reflected from the object 20 into thereceiver 610 and detector 612. This data can be used to providecorrections to the range information as well as the information on thematerial and/or finish of the surface of the object 20.

Digital Signal Processor

A digital signal processor integrated circuit controls all the timecritical functions of the FDV—scanner control, laser firing. It alsoprovides fast floating point computation capability for making geometrycorrections, calibration corrections, and video lens corrections, andvideo compression. The digital signal processor is interrupted atregular time intervals, typically about 10 usec. At each of these timeintervals, a check is made to see what real time calculations areoutstanding.

Scanner Control

The electronics for the scanner are a simple precision PID controllerwhich are driven by a digital signal from the DSP. When driving thissystem quickly, there is noticeable lag in the ability of the scanner tofollow the driving signal. However, the controller circuit does not havean error signal output. An external precision analog differentialamplifier provides an error signal (the difference between the commandsignal and the actual displacement), which is sampled by the DSP at lowresolution. The DSP then computes the exact scan position by computingthe sum of the command signal and the error signal. The advantage ofthis method is that it requires only a low resolution AID converter anda precision D/A converter, rather than a far more expensive precisionA/D.

The digital signal processor generates the trajectories for the analogscanner controller, and makes measurements of the difference between thedesired trajectory and the actual position. It predicts the time atwhich the laser pump is turned on so that the laser will fire at thedesired angle. These predictions are made at regular time intervals.FIG. 10 is a flow chart that shows the calculations performed at eachtime interval.

Trajectory Computation

The user defines areas within the view of the scanner that are to bescanned, and indicates the density of points to sample within thescanned region. There are several scan patterns which can be used, andthese require a specific pattern of mirror movement, known as thetrajectory. The objective of picking a good trajectory are theconflicting needs of doing the move quickly and accurately. Accuratemovement requires minimum torque, which would otherwise deform theapparatus. This limits the speed with which motions can be made. Atequal time increments, a calculation is performed to determine thecurrent position of each mirror. The particular calculation used dependsupon the type of scanning employed.

Raster Scanning

When the desired scan field is a polygon, one of two raster scanningpatterns is used. In the first, scanning is uni-directional (i.e.,always proceeds from left to right, or right to left, on parallellines). FIG. 11A shows such a unidirectional scan pattern. In betweenscan lines, the scan mirror retraces to the beginning of the next linewithout making any range measurements. The retrace can proceed quitequickly since no measurements are being made during the retrace.

A slightly more efficient means of raster scanning is bi-directional inwhich scanning is also performed on the retrace. FIG. 11B shows such abi-directional scan pattern. This is not as efficient as it might seembecause the retrace time is used for other calculations, and because theresulting scan pattern is not as regular.

Both raster scanning methods require traversing a straight line in theminimum time, starting at zero velocity and ending at zero velocity. Thetorque applied to the mirror is proportional to the angularacceleration, which must zero at the beginning and end of the scan sincethe mirror is at rest. It can be shown that the trajectory that makessuch a minimum energy move between two points is given by the sum of astraight line and a full cycle of a sin. However, this is closelyapproximated with much less computation by the minimum degreepolynomial, with boundary conditions p(t0)=p0, p′(t0)=0, p″(t0)=0,p(t1)=p1, p′(t1)=0, and p″(t1)=0 which is the fifth order polynomial:p(t)=(p₁−p₀)t′³ (6t′²−15t′+10)+p₀, where t′=(t-t₀)/(t₁−t₀).

Spiral Scanning

A disadvantage of raster scanning is that since the speed of thetrajectory is varying, the scanning efficiency is not optimal. A spiralpattern can achieve a constant speed trajectory which permits a uniformpoint distribution.

Seeking

In addition to scanning a range image, the system is capable ofperforming a number of functions which are common in surveying. Thescanner can be made to search for important features, or locations ofhigh reflectivities. This allows the system to perform normal surveyingfunctions by finding a target whose location is approximated identified,and reporting its exact angles and position.

Angle Calibration

The capacitive encoders in the moving coil motors have tremendousrepeatability, but relatively poor accuracy. A number of calibrationactivities need to be continuously performed to ensure system accuracy.

Before use, each scanner is calibrated over its complete range ofangles. At a number of discrete temperatures, a map is created andstored of the measurements of apparent angles for thousands ofaccurately measured points using an external resolver that is traceableto NBS standards. The DSP linearly interpolates between these measuredpoints on every angle measurement.

Preferably, the accuracy of angle measurement is improved by determiningscale or offset errors in the encoder during operation. Commerciallyavailable scanners can drift significantly with environment changes.This results in a shift in the effective zero and full scale range ofthe angle measurement, while maintaining the overall shape of thecalibration curve obtained by making careful laboratory measurementsbefore the scanner is installed in the system. The environmental effectis reduced by providing a means for determining when the scanner is at aknown and repeatable angle. In one preferred embodiment of such asystem, two optical references which are fixed with regard to the caseof the instrument are aimed at the back of each scanning mirror. Thereare a variety of mechanisms for providing the optical reference, but inone preferred embodiment, a pair of autocollimators are aimed at areflective surface on the back of the scanning mirrors and will providea highly repeatable measurement of when the mirror is normal to the axisof each autocollimator. Each autocollimator gives a reference angle towithin approximately 10 μrad. Periodically, the scanner is moved undercomputer control to the position at which the mirror is closes to beingnormal to the autocollimator axis, and the apparent angle is measured.The measurements are compared with the measurements taken when thescanners were calibrated, and a linear correction is calculated andapplied to every subsequent measurement.

In an alternative embodiment, a pair of mechanical stops is providedjust past the normal range of motion of the scanning mirror.Periodically, the mirror is driven until it touches a mechanical stop.Then, the scanning mirror is driven with a known current, whichcorresponds to a known force. The mirror arrives at equilibrium at avery repeatable position, and this is used to calculate a linearcorrection to the mirror calibration curves.

Range Calibration Fibers

The timing circuits have a certain amount of offset and scale drift withtime and temperature, and a provision has been included to compensatefor these variations. When an optical pulse is emitted from the laser602 a small amount of the energy is sampled by a beam splitter 810 andintroduced into a single mode optical fiber 830 by focusing the beamusing a lens 833 on the fiber face 831. The other face of the fiber 832is arranged so that the beam which comes out of it is collimated into abeam which enters the lidar receiver 610. The fiber can either beproduced so that its length does not-vary with temperature, or itsvariation in length with temperature can be accurately characterized.When a single mode fiber is used, the variation in propagation delaywill be less than a few picoseconds and the pulse shape emitted by thefiber will be nearly identical to the pulse shape going into the fiber.Periodically, the timing circuits are used to measure the propagationdelay through the fiber, and corresponding adjustments are made to therange measurements taken from external surfaces.

The fibers can be manufactured so that the end at which the pulse islaunched 833 and from which the pulse is emitted 834 are partiallyreflecting. When this is done, the pulse enters 833 and is propagated tothe opposite end 834, at which point only some of the energy is releasedand the rest returns to the first end 833. Again, a fraction of thelight is emitted, and the rest reflected, which eventually is emittedinto the receiver is process repeats until the remaining energy in thefiber falls to negligible levels. The result is a sequence of pulses,commonly 3-10, being applied to the receiver, which have delays allrepeatable to within a few picoseconds. Periodically, the timingcircuits are used to measure these pulse trains from the fiber, andcorresponding adjustments are made to the range measurements taken fromexternal surfaces.

Range Walk Calibration

The lidar system measures the range of surfaces by timing the delaybetween the laser pulse being emitted and returning from the surface.This delay is measured electronically by imposing a sample of theoutgoing pulse, and the return pulse on an optically sensitiveelectronic detector 612 embedded in the receiver 610. In one,embodiment, the electronic timing circuit measures the time between whentheoutgoing pulse exceeds a set threshold voltage, and when the returnpulse exceeds the same voltage. The outgoing pulse will be the sameintensity within a few percent. However, many surfaces vary greatly inthe amount of light that will be reflected. The result is that theapparent relative time for two pulses which occur at the same range buthave different intensities may appear to be at different ranges. Themeasured time for a small pulse 3810 to first exceed the threshold levelwill be later than the measured time for a large pulse 3830 to exceedthe same threshold, even when the pulses return from objects at the samerange. Thus highly reflective objects or objects at distances of maximumtransceiver sensitivity will appear slightly closer. This creates anapparent “range walk” as a function of intensity. The range walk can becorrected if the shape of the optical return is always the same and theenergy of the return is known. The extremely repeatable shape of thepulses generated by the passively Q-switched microchip laser makes thispossible.

Part of the timing circuit estimates the energy in each detected pulse.A table of corrections is maintained to improve the range estimates. Twodifferent circuits have been employed to make a measurement of the pulseenergy for this purpose. The first is a gated integrator, the gate beingopen at the beginning of the pulse, and closed at the end. The signal isapplied to a comparator 3920 which closes the switch 3930 when thesignal exceeds a selected level, and closes it when the signal fallsbelow the same level. The signal is also applied to a delay 3910, andthe output of the delay goes through the switch 3930 when it is closed,and is applied to the integrator 3940 over the period of time the switchis closed. The delay is chosen to compensate for the time lag in thecomparator and switch. When the pulse is complete, the value of theintegrator is sampled by an analog to digital converter 3950. The secondconsists of a integrator with a time constant scaled to the width of thepulse 4010, followed by a peak detector 4020 which has a time constantmuch longer than the pulse width. The output of the peak detector issampled shortly after the pulse is detected.

Periodically, the timing circuit is used to measure a sequence of pulseswhich have been delayed by the single mode fibers 830 used to calibratethe offset and scale factors associated with the time circuits.Additionally, the intensity of these pulses are varied over a broadrange by a variable attenuator 820. By altering the amount of lightcoupled into the fiber, the energy of the detected pulses can be variedover the dynamic range of the receiver, at one particular time delay.The intensity and the measured time delay values produce a map of therange walk correction required for each intensity, and this correctionis applied to subsequent measurements. This correction can provideaccuracy of 1 mm over the dynamic range of the instrument, particularlyas a result of the great repeatability of the laser pulse waveform. Thisfunction is then used to correct the measured range of external surfacesas a function of light intensity returned from those surfaces.

Geometry Calculation

The output of the FDV after a range scan consists of points in sphericalcoordinates with respect to a coordinate system in the scanner. However,the raw data consists of mirror angles and time intervals. The DSPcomputes the spherical coordinates of the scanned points by taking intoaccount scanner geometry (mirror thickness, optic axes, mirror offsets,etc.) and all the appropriate calibration adjustments.

Laser Control

Delay Prediction

The digital signal processor is responsible for controlling the firingof the pulsed laser, but it can only do so indirectly. The processor hascontrol of the timing for staring the pump diode, which causes thepassive q-switch to fire after saturation has occurred. However there isa variable delay between turning on the pump and having the laser fire.The delay is a function of junction temperature, which in turn is acomplex fiction of ambient temperature and recent history of laserfiring. The delay generally ranges between 100-300 usecs.

Fortunately, it is primarily necessary to know the scanning mirror angleat the precise moment the laser fires. After the laser has been firedjust a few times, the pump delay does not change quickly if the firingrate does not change quickly. As a result, accuracy of a fewmicroseconds can be achieved by estimating the next pump delay to be thesame as that in the previous firing cycle. The digital signal processormeasures the pump delay by reading an internal counter when the pump isstarted and when the laser actually fires, causing an interrupt. Sincethe interrupt latency is less than a microsecond, this becomes thetiming accuracy to which the pump delay can be measured.

A more sophisticated dynamic model of the thermal properties of thelaser could lead to slightly enhanced scanning pattern regularity, butis probably equally limited by the time resolution of the processorinterrupts.

Firing Control

Given a time vs. angle trajectory for a scanning axis, w(t), a desiredangle to fire the laser, and an interrupt interval Dt, the decision tofire the laser amounts to computing the time at which point the pumpdiode is started.

Computer Control

The FDV is designed to perform under the control of a remote hostcomputer which contains graphical controls for a user to specify areasto be scanned. The remote machine controls the FDV through abi-directional serial byte stream, which is effected in any of a numberof media: Ethernet, EPP parallel port, serial port. A processor in theFDV is assigned the task of decoding messages, and scheduling therequired activity. FIG. 12 is a block diagram of the FDV processor.

Host Communications Interface

The host machine acts as a master, sending a well defined messageprotocol to command the FDV. When actions are completed, the FDVresponds with data and status information. Among the actions which canbe requested are:

Point the scanner

measure a distance

range scan a box

fire the laser n times

take a video image

Scanner Control

Referring to FIG. 13, in normal operation, each scanner in the dualmirror system requires a 16 to 18 bit digital word to set the desiredposition, which is applied to a precision digital to analog converter tocreate a voltage proportional to the desired position. However, therewill be some error between the position commanded by the output of thisconverter and the actual position of the scanner, which is reflected bythe output of the position encoder. A precision difference signal isgenerated, and the difference is measured to 12 bit accuracy. Thisprovides an economic method of making 18 bit position measurements whileonly using an inexpensive 12 bit converter.

Commercially available galvo scanners have microradian repeatability,but have relatively poor scale and offset performance, particularly overtemperature. A calibration mode has been incorporated into the system topermit making measurements at two precise angles, and using the twomeasured data points the offset and scale drift of the scanner can becalculated.

Two methods have been developed for this purpose: an optical and amechanical means. In the mechanical method, the scanner shaft is gentlyplaced against one of two mechanical stops, and the current in thescanner controller is adjusted to a specific value, which provides aknown force. The position signal is adjusted until there is no positionerror, and this gives the calibrated position measurement. In theoptical method, two autocollimators are aimed at the back of the scannermirrors, which have also been polished and mirror coated. When thescanner mirrors are exactly aligned with one of the collimators, theoutput from the split photodetector in the autocollimator is balanced.By placing the scanner in each of these precise angles in turn, anoffset and scale correction for the scanner encoder can be calculated.

Timing Circuit

The purpose of the timing circuit is to provide the relative timebetween the start pulse and the stop pulse, in picoseconds. There aretwo subsystems in the timing circuit: a signal conditioning and energyintegration circuit (an embodiment of which is shown in FIG. 14), and atime interval analyzer. Both communicate directly with the DSP.Initially, systems have been produced with a commercial timinginstrument, the Stanford Research Systems SR620 time interval analyzer.The interface to this instrument is through an IEEE488 interface. In apreferred embodiment, the communications interface to the StanfordResearch Systems SR620 time interval analyzer is IEEE488.

A custom time interval measurement circuit has been developed whichutilizes a separately patented interpolation technology. The circuitemploys a clock, typically operating at >100 mhz, which is used to makea coarse count of 10 nsec intervals between stop and start pulses.Additionally, there is an interpolator which divides each 10 nsec coarsecount into 1000 smaller increments, providing 10 psec resolution. Thissystem has approximately 5 psec jitter. Differential time measurementscan be made with less than 20 psec RMS error, which corresponds to about3 mm. This circuit communicates with the DSP using a dedicated serialbus, and employs a packet protocol: the DSP arms the circuit by sendinga single byte. When the timing circuit completes its task, it sends asequence of bytes which represent both the time delay between start andstop pulses, and the intensity of each pulse.

Laser Firing

The DSP has three lines for laser control: one starts the laser pump,the second indicates that the laser has fired, and the third indicatesthat the return pulse from a target has been detected. When the laserfires, the DSP samples the analog pulse amplitude signal. This happenstypically within 1 μsec.

Video

For targeting, the user is provided on the host a video representationof the scene from which he can choose a portion to be range scanned. Inmost cases this will correspond to the scene rendered in ambientillumination.

Capture

One way the video is captured is by using the scanner to aim singlesensitive detector across the scene with the laser turned off. Thispermits acquiring an image which has very accurate spatial alignmentwith subsequent range scans. However, image capture can be quite slow incomparison to commercially available cameras.

A second approach is to utilize standard commercial CCD video cameras toacquire an image. One CCD camera with a wide angle lens is aligned withthe range scanner with as small an offset as possible. A second camerawith a 5 degree field of view is placed so that its optic axis iscoaxial with the transceiver. Thus, a much smaller field of view isaccessible through the scanner, and can be scanned with the sameresolution as the transceiver. This allows targeting small or distantobjects.

Alignment

The wide angle lens introduces a fish-bowl effect in the image capturedby the CCD sitting behind the lens. Straight lines in the world are notstraight in the image. This distortion increases with the distance fromthe center of the lens. This distortion is removed by comparing theimage the camera produces when aimed at a carefully designed and printedcalibration target image. The difference in the anticipated image andthe recorded image provides the information needed to warp subsequentlyacquired images to eliminate the distortion.

Compression

Each video image is compressed prior to transfer. Currently we are usingJPEG standard image compression. It is relatively fast, and createsreasonably small compressed images for communication. Another desirablefeature is that the algorithm operates on blocks, which permits us to dointerleave image capture, alignment, compression, and transmission inparallel—significantly enhancing throughput.

Point Video

A second camera, with a narrow field of view (e.g., approximately 5°) isplaced such that it is coaxial with the scanning laser beam. The fieldof view is adjusted so that the pixel resolution is approximately thesame as the voxel resolution of the lidar system. The camera can beoperated while the laser is activated. When this is done, a small groupof pixels will be illuminated by the laser, and the centroid of thesepixels will correspond to the point which would be measured by thelidar. When a video image is captured, it can be mapped onto a surfacewhich is estimated by a lidar scan.

Computer Graphics Perception (CGP) Software

Referring to FIG. 15, the CGP 40 is a software system that runs on a CGPComputer 1500 and communicates with the FDV 10. The CGP 40 runs on manydifferent types of computers, including laptops and workstations. TheCGP 40 operates on a computer 1500 with a suitable display device 1510,such as a color graphic display terminal, a suitable character inputdevice 1520, such as a keyboard, and a suitable pointing device 1530,such as a mouse. The software can use any number of standard 3-Dgraphics rendering libraries to interactively present the acquired 3-Ddata in a window on the display device. The portion of the CGP 40 userinterface that involves 3-D view manipulation and data projection into awindow is handled by the 3-D library.

The CGP 40 performs real time 3-D data acquisition and modeling in thefield. The CGP's 40 functionality includes high level FDV 10 control,targeting and data acquisition; display and visualization of scannedpoints; surface segmentation and fitting; manual 3-D model construction;3-D visualization; interaction with part and model databases; and theability to export data in standard data exchange formats to other CADsystems for further processing. The integration of hardware andsoftware, as described here, enables major improvements in productivityand quality in the overall process of three dimensional modeling ofreality.

With reference to FIG. 1A, the data acquisition and modeling processdivides into the following steps: FDV 10 control, point acquisition,segmentation, geometry fitting, modeling by manipulating the geometry,scene registration with or without warping, model annotation, andgeometry display and query.

With reference to FIGS. 16A and 16B, the foregoing operations may beperformed in at least two graphic display windows. One window 1610 (FIG.16A) displays a video image of the target scene used to define regionsto be scanned by the FDV 10 while the other window 1620 (FIG. 16B)displays an interactive 2D projection of the 3-D model consisting of thescanned points and constructed surface geometry as well as otherinformation about the scene. Additional windows may be used to providemultiple views of the data. In addition, the CGP 40 provides additionalwindows for controlling the FDV 10 hardware and for setting anddisplaying the status parameters of the system.

Scan Control

Referring to FIG. 15, prior to using the integrated hardware/softwaresystem the FDV 10 is positioned to point in the direction of the objects20 of interest.

Scan control is the process of indicating which portions of the scenethat are visible to the scanner are to be scanned. Different parts ofthe visible scene can be scanned at different densities, since simplegeometric objects, such as planes, cylinders and spheres can beaccurately modeled with a fairly low number of scan points. Therefore,the region in front of the scanner is often captured in multiple scans,rather than in one high resolution scan. Only regions with high levelsof detail need high resolution scans.

Referring to FIG. 17A, one of the means of scan control is the use of avideo image 1710 of the scene acquired from the FDV 10. Using a pointingdevice, such as a mouse, one can indicate the region to be scanned byany number of methods, such as dragging out a rectangle 1720 on thevideo image. The COP 40 instructs the FDV 10 to measure the range ofwhatever object exists at the center of the user specified targetingregion to assist in specifying the scan density, since the angle betweenpoints is determined by both the desired density on the surface and thedistance from the scanner. A means for specifying the desired scanparameters, such as a dialog box, is provided and allows the user tospecify the scan parameters in a variety of ways, including pointdensity, point spacing, or total number of points in each of thevertical and horizontal directions.

The CGP 40 then translates the region and scan resolution informationinto a set of commands for the FDV 10. These commands are communicatedto the FDV 10 using a means of communications, such as a TCP/IP networkconnection, and the acquired data is also returned to the CGP Computer1500 using the same means.

Additional scans at different densities can be initiated in the sameway, or one can use previously scanned data points rather than the videoimage to specify new scan regions. If the view of the scanned data isoriented so that it is exactly aligned with the scanner direction, thena scan region can be indicated by methods such as dragging out arectangular box. When the data is aligned to the scanner in this waymost of the 3-D information is difficult to see, therefore, the softwarecan display the points with the intensity of the returned laser light ateach point color mapped as described in the next section. The intensityinformation is often sufficient to identify objects in the data window,so that new scan regions can be defined. Alternatively, the user canmodel and/or color some of the objects in the scene to help locateregions of interest in the window. Using the data window to define newscan regions avoids any parallax errors, since the view is aligned withthe scanner.

Scan control can also be achieved by using the pointing device to movethe laser beam and highlight points in the actual scene. Any number ofmethods could be used to describe the desired scan region by moving thelaser beam and identifying points of interest by a user action, such asclicking a mouse button. Methods could include operations such as:indicating a bounding box by moving the laser to diagonally oppositecorners of the desired scan regions; indicating the top, bottom, leftand right bounds of the scene; indicating a sequence of points thatrepresent the bounding polygon of the scan region; indicating the centerof the scan region and using other means, such as dialog boxes todescribe the extent of the desired scan region.

Point Acquisition

With reference to FIG. 17B, the data returned by the FDV 10 consist ofthe coordinates of the points and their intensity values. In onepreferred embodiment, the scanning is performed in such a way that thedata returned lies in an ordered grid of three dimensional points 1730.Viewed from the scanner, these points appear as a regular rectangulargrid, much like a bitmap. However, each point consists of itscoordinates in three-space as well as the intensity of the reflectedlaser pulse at that location.

Each point returned is displayed in the data window 1620 as it istransmitted by the FDV 10. The CGP 40 lets the user interactively changethe 3-D view of the data while the data is arriving to get a better ideaof the spatial layout of the data. Also, to help visualize differentfeatures in the data, of the CGP 40 can allow each point to be colormapped from the intensity of the reflected laser pulse at that location.A scan cloud 1810 is shown in FIG. 18 representing the surface of ahorse sculpture. Instead of having all the points in a single color, asshown in FIG. 18, one can map different laser return intensity values todifferent colors, and produce a multicolored scan field 1910 as shown inFIG. 19. The intensity color mapping provides considerable extra surfacefeedback to the user and this is useful in both targeting and modeling,as described later.

The ordered grid of points generated is referred to as a scan field.Multiple, possibly overlapping scan fields may be gathered andsimultaneously displayed in the manner described above. The datastructures within the CGP 40 maintain the list of scan fields, so eachdata point is always associated with a scan field. The scan fieldsusually contain data points from the surfaces of many different objects,so they need to be partitioned into smaller groups of points, asdescribed in the next section.

Segmentation

Segmentation is the process of grouping together points that werescanned from the surface of the same object. The points from a singleobject may be a small portion of a scan field, or may occur acrossmultiple scan fields. The segmentation process may be manual, asdescribed below, or automated, as described later in theauto-segmentation section.

Referring to FIG. 20, the first step of the manual segmentation processis to select one or more scan fields 2010 that contain scan points onthe object of interest. Selecting one or more scan fields can beperformed by any of the conventional means, such as using a pointingdevice, possibly together with keyboard keys. Selecting a scan fieldselects all of the points in the scan field. The group of pointsresulting from this step form a pool of candidate points that can now betrimmed to remove points on other objects. Each point in the pool isinitially marked as selected, and the operations described below can beused to toggle the point states between selected and deselected.

Referring to FIG. 21, scan points 2010 from a desired object surface canbe cut out using one or more lasso operations from possibly differentviews. The user can manipulate the view direction as needed to give aclear view of the desired subset of the point pool. The user then usesthe pointing device to draw a polygonal lasso region 2110, which dividesthe screen into two regions: the interior and exterior of the lassopolygon. The following operations are supported: mark all points in aregion as selected and all of the other points as deselected, mark allof the points in a region as selection without affecting the otherpoints, and mark all of the points in a region as deselected withoutaffecting the other points. The lasso operation may be repeated as manytimes as necessary to refine the selection, possibly changing the viewof the scene between lasso operations. The user can the cut thecurrently selected set of points out to form a new point set. The newpoint set acts like a scan field in that it is, and can take part in thefitting operations described in the next section. In FIG. 22, three newgroups of points 2210, 2220 and 2230 have been created using the manualsegmentation method described here, and some points near theintersection of the planes are left from the original cloud of points.

Geometry Fitting

In one preferred embodiment, the CGP 40 can contain many geometricprimitives that can be used to simulate the actual surfaces of theobjects scanned. The geometric primitives include any number of standardgraphics primitives, such as triangulated meshes, planes, cylinders,spheres, torii, lines, and points. The simplest form of geometry fittinginvolves using a triangulated mesh to connect the scan points to showthe surface features of the objects scanned. The scan cloud 1810 in FIG.18 can be meshed 2310 and rendered as shown in FIG. 23. Since the scandata are acquired in a regular grid, it is simple to create a triangularmesh by connecting neighboring points. The user can also setdiscontinuity tolerances in depth and angle to avoid meshing adjacentpoints separated by more than the specified threshold. Breaking the meshin this way provides a more realistic looking surface, referred to as ashrinkwrap surface, because artificial mesh surfaces at occlusion edgesdo not occur. A wide variety of known mesh operations can be applied tothe resulting mesh, such as smoothing (noise reduction) and meshsimplification (to reduce the mesh density in smooth areas that do notrequire a fine mesh grid). Mesh vertices may also be colored withinformation such as intensity.

As stated above, the CGP 40 includes many standard geometric primitives.Before fitting the points to such objects, the point clouds must besegmented as described above. Once segmented, each group of pointsrepresents a single surface that can be fit by a geometric object Thefitting can be guided by the user, who may know the type of shape to befit. For instance, after scanning the corner of a room it is clear tothe user that the points on a wall can be fit by a plane and the pointson a pipe can be fit by a cylinder, so the user can request the fit of aspecific object It is also possible to semi-automate this process toidentify which shape best fits a particular point group.

Fitting a plane to a set of points is a simple problem that has manywell-known solutions. The extent of the patch used in the CGP 40 torepresent the plane can be determined by the convex hull of the pointsin the plane. For instance, the three point groups 2210, 2220 and 2230shown in FIG. 22 can each be separately fit to planes 2410, 2420 and2430 as shown in FIG. 24 using any available fitting algorithm.

Many standard approaches are available for fitting more complex shapes.In one preferred embodiment, two phases are involved: a parameterestimation phase to get a starting point, and an optimization phase,where the parameters are varied to minimize the overall error. The totalerror is the sum of the squares of the distance between each scan pointand the nearest point on the surface of the object being fit. Theoptimization phase uses conventional optimization methods to reduce theerror between the object, as defined by its parameters, and the datagiven by the scan points.

A cylinder fitter can convert a cloud of points 2510 as shown in FIG. 25into a cylinder object 2610 as shown in FIG. 26. All fitted objects,including the cylinder, reference the original points that were used tofit the object. The user may choose to view the resulting cylinder 2610or the original points 2510, or both, at any time. Using manual orautomatic segmentation methods, it is possible to convert the scanclouds, 2710 in FIG. 27, representing many cylinders into the best fitcylinders 2810 shown in FIG. 28. Once each cylinder's diameter and axisare established, it is possible to manually or automatically add elbows2910 in FIG. 29 to complete the modeling process.

A cylinder is described by five parameters: a normalized vectordescribing the cylinder axis (two independent parameters), the radius,and two additional parameters that are used to locate the line of actionof the cylinder axis in space. The length of the resulting cylinder canbe determined by projecting the scan points onto the cylinder axis andnoting the extreme values of this projection.

Two novel methods for estimating cylinder parameters are implemented inone preferred embodiment. The first way to find initial parameterestimates for a cylinder is to find approximate surface normals, asdescribed in the auto-segmentation section. If all of the normals areset to unit length, then they can all be consider to be vectors from theorigin to a point on the surface of the unit sphere. If one uses eachnormal vector and its to accumulate a group of points on the unitsphere, then one can fit a plane through the resulting group of points.The resulting plane normal is roughly parallel to the cylinder axis.Given the cylinder axis and the plane from the previous step, one canproject the scan points onto the plane. The projected points will bewell described by a circle in this plane, since the plane is normal tothe cylinder axis. A best fit circle can be calculated using theprojected points on the plane to give an estimate of the cylinderradius. The center of the circle on the plane can be converted to a 3-Dpoint to give a point on the cylinder axis.

The second way to estimate the cylinder parameters is to fit the set ofpoint to a quadric surface, which is described by the implicit equation:

F(p)=0=c ₁ p ₁ ² +c ₂ p ₂ ² +c ₃ p ₃ ² +C ₄ p ₁ p ₂ +c ₅ p ₁ p ₃ +c ₆ p₂ p ₃ +c ₇ p ₁ +c ₈ p ₂ +c ₉ p ₃ +c ₁₀  (1)

where p={p₁, p₂, p₃} is a point on the quadric surface.

One can then take c₁₀=−1 since the equation is implicit and perform aleast squares fit with all of the data points to determine the othernine parameters. After determining the best fit quadric surface for agiven set of points the next step is to find a point actually on the newsurface (p_(s)) that is in the vicinity of the other points. This isachieved by finding the centroid of the scan points (p_(c))and thenfinding the closest point on the surface of the quadric to give p_(s).The normal to the surface at point p_(s) can be determined using:

N _(p) =D ₁ p _(s) +D ₂  (2)

where ${D_{1} = \lbrack \quad \begin{matrix}{2c_{1}} & c_{4} & c_{5} \\c_{4} & {2c_{2}} & c_{6} \\c_{5} & c_{6} & {2c_{3}}\end{matrix}\quad \rbrack},{D_{2} = \begin{bmatrix}c_{7} \\c_{8} \\c_{9}\end{bmatrix}}$

Two unit vectors, u₁ and u₂, are then found such that they are normal toboth each other and N_(p). These vectors form a basis for the surface atthe point under consideration, and additional vectors on the surface canbe found as:

v _(α) =u ₁ cosα+u ₂ sinα, 0≦α≦2π  (3)

The unit principle vectors v_(α) are then found by determining therotation a that satisfies:

v _(α)·(N _(p) ×D ₁ v _(α))=0  (4)

There are two solutions to this equation that yield the orthogonal unitprincipal vectors v₁ and V₂. The surface curvatures κ in these twoprinciple directions are then given by: $\begin{matrix}{{\kappa ( v_{i} )} = {{- v_{i}} \cdot ( \frac{D_{1}v_{i}}{N_{p}} )}} & (5)\end{matrix}$

For cylindrical surfaces one of the principal curvatures will be nearzero, and the radius of the cylinder is the reciprocal of the absolutevalue of the nonzero curvature. A method to determine the radius (r) andaxis of the cylinder has been described, and only the location of theaxis needs to be determined. A unit surface normal can be calculated as{circumflex over (n)}=N_(p)/∥N_(p)∥. The sense of the normal can beadjusted so that it points towards the interior of the cylinder byensuring that {circumflex over (n)}·(p_(c)−p_(s))>0 since the centroidof the points lies on the interior of the cylinder. A point on the axisis then given by p_(s)+r{circumflex over (n)}. These starting parameterscan then be used in a minimization process to find the best fitparameters for the cylinder.

The novel method described above for curvature estimation using thequadric surface formulation is further used in a novel way for automatedobject type determination. If the points being fit are well representedby a plane then both principal curvatures will be near zero. If thepoints being fit are from a cylindrical surface then one curvature willbe near zero and the other will be nonzero. If the points are from asphere then both curvatures will be nonzero and their magnitudes will beapproximately equal. Combining the automatic detection of object typeand the auto-segmentation algorithm, described later, allows the CGP 40to have a novel method for automatic fitting of many objects that occurin typical scanned scenes.

A further use of the curvature estimation is sphere fitting, which isachieved by using the quadric surface approach to approximate the radiusand location of the center point, and then using a four parameter(center point and radius) minimization to reduce the error between thesphere model and the measured points. The novel method described abovefor finding a point on the axis of a cylinder is also used in thepreferred embodiment to find the center of a sphere.

The segmentation techniques disclosed above can be used to create avariety of useful fitting tool based on combinations of the previouslydescribed shapes. For instance, a corner, consisting of an intersectionof three planes which may or may not be orthogonal, is a very commonfeature to scan. Knowing that the specified point group contains threeintersecting planes, such as the points 2010 in FIG. 20, the points areautomatically segmented, using the technique described later, into threesubgroups of points that each lie on separate planes. Then, anyavailable plane-fitting algorithm can be used to fit the planes to thescan points in each group. Unlike the more general auto-segmentationalgorithm described later, if one knows that the corner object consistsof three planes, the fitting algorithm need not try to fit cylinders,spheres or other objects and check which produces the best fit, onlyplanar fits are required. Referring to FIG. 30, the corner fitting toolnot only fits the planes 3010, 3020, 3030 but can also intersect them tocomplete the corner. As shown in FIG. 30 additional useful informationcan be given to the user by introducing lines 3040, 3050, 3060 at theintersection of pairs of planes, and a vertex point 3070 that representsthe location of the corner. This vertex point is much more accurate thana single scan point, because each plane is fit using many data points,and the vertex is generated by intersecting the planes. The above novelmethod for automatically creating a corner with its intersection linesand vertex can be used as a tool of the CGP 40.

Each object stores information about the quality of the fit, so that theuser can query the object and examine the mean, standard deviation, andworst errors. Knowing the accuracy of the FDV 10, the CGP or the usercan then decide if an error has been made during the fit. Errors canarise when the wrong type of primitive is fit, or when extraneous pointsthat were not actually scanned from the desired surface remain in thedata set. In addition, the objects store their geometric parameters sothe users can query for radius, length or other values of interest.

In addition to the class of general object fitters, which are givenclose to no initial information other than the points to fit, there is aclass of fitters that can take advantage of foreknowledge about objectsin the scene. An area in which such foreknowledge exists is that of theconstruction industry, where parts used follow standards of dimensionsand design. For example, the external diameter of pipes from aparticular manufacturer may come in five different sizes: 4″, 5″, 6.5″,8″, and 10″. This information typically resides in tables that describerelevant attributes of these parts. The cylinder fitter can takeadvantage of the information in these tables to significantly reduce thesolution space to be searched: the fitter need only search for solutionsinvolving cylinders of one of those diameters. Another way to use suchtable lookups is to have the fitter come up with a general solution,then match against entries in the object tables to find the entry withthe closest parameters. For example, a pipe fit by a cylinder with 7.8″diameter would be matched against the 8″ entry in the table from theexample above; the user (or fitter) then has the option of refitting an8″ cylinder to the pipe, or accepting the 7.8″ cylinder. Yet another useis for the user to manually select a specific entry (or collection ofentries) from the table and tell the fitter to use its parameters in thefit, which also reduces the fitter's solution space (which can decreasethe time taken).

Modeling

The fitting of geometric primitives, as described in the previoussection does not usually complete the modeling process. It is often thecase that only a portion of the objects surface is scanned, such as oneend of a cylinder or a portion of a wall, and further operations arerequired to complete the 3-D model. Modeling is the process ofcompleting the construction of the 3-D model given some fitted geometricprimitives.

Many common CAD operations such as extension, intersection (mutualextension) and trimming are available in the CGP 40. For example, thecylinder 3110 in FIG. 31 does not initially extend to the plane of thefloor 3120. In FIG. 32 the cylinder 3220 has been extended to the floorplane 3120. These simple operations enable rapid completion of portionsof the model from the objects created by geometry fitting. For instance,given three planar patches that were fit from scan data near a corner,it is easy to mutually extend the three planes to complete the cornerfeature.

Object extensions can be accomplished in several ways. One way is toselect the geometric object to be extended and tell it to extend to asubsequently selected object. The nature of the extension is determinedby both the type of object to be extended and the second objectselected. For example, a cylinder extends the end closer to the secondobject along its centerline until its end intersects with the infiniteplane defined by the second object's geometry (in the case of a planarpatch, the infinite plane is that of the patch's plane, and for acylinder, the infinite plane is that containing the centerline and asorthogonal to the extending cylinder as possible).

Another way is to make use of object handles, which are nodes that theuser can grab. These handles are tied to an object's definition(position, orientation, and size) where appropriate, and by moving ahandle, the object's definition changes accordingly. Again, taking thecylinder as an example, the same extension described above can beaccomplished by grabbing the handle on the end to be extended, and thenmoving the handle (and extending the cylinder) to the desired position.A handle's motion depends on the part of the object with which it istied; a handle on the centerline of the cylinder is constrained to moveonly along that centerline, while a handle on the boundary of a planarpatch is constrained to move only inside the plane of the patch. Forsome objects, handles may be inserted and removed, changing thedefinition of the shape of the object (for example, handles on a planarpatch have a one-to-one correspondence to vertices on the planar patch'sboundary). Other handles can provide rotational control over an object.The control of handles is interactive and dynamically updates, so theuser can see the intermediate results of the redefinition.

A new operation, called merging, has been developed to allow differentportions of a single object surface to be joined to form a single objectin the CGP 40. It is often the case that one's view of an object isobscured by other objects in front of it. For instance, the view of aback wall in a room may be divided into two pieces because of a columnin the foreground. A scan of the region will result in different groupsof points on the particular object. If auto-segmentation is used, asdescribed later, rather than manual methods where a user knows that thepoints belong to the same object, then separate point groups would beformed. Each point group would then be fit to a separate object,resulting in multiple pieces of the same surface. The two objects in theCGP 40 that are known to be on the surface of the same feature, such asthe two planar patches of wall obscured by a column, can be merged toform a single object. Each object stores a reference to the data pointsdefine it, so when a merge request is received a new geometric fit isperformed on all the underlying data points that were part of theconstituent geometries to achieve the best overall fit This novel methodof increasing the accuracy of object fitting is used in the mergingoperations of one preferred embodiment of the invention. Thecharacteristics of the two primitive objects that were merged do notaffect the outcome of the merge; only the underlying point positions areconsidered.

Using the manual or automatic methods, the user can take a cloud ofpoints 3320 from the surfaces of many objects, such as the points on thepyramid 3310 shown in FIG. 33 and convert them to a set of geometricobjects 3420, such as the planar face of the pyramid 3410 in FIG. 34.The modeled scene shown in FIG. 34 accurately represents the features ofthe original objects that were scanned, and allows measurements to bemade between any particular locations in the modeled scene.

Scene Registration

The initial position of each scan point is described in a localcoordinate system whose origin is that of the FDV 10, and whose axes arefixed relative to the FDV 10. Therefore, scan fields taken withoutmoving the FDV 10 are inherently registered in that all the scan pointsuse the same coordinate system. However, if the FDV 10 is moved betweenscanning operations, additional effort is required to transform the datato use a shared coordinate system. This is a six parameter rigid bodytransformation of a data set, involving three translation and threerotation parameters, and is easy to apply to the data once thetransformation is known.

A novel process is used to register the scan fields from different FDV10 positions. The novel registration process requires the user toidentify pairs of points, lines or planes in two different scenes thatrepresent the same feature. It is also possible to use differentfeatures, such as the back and front face of a wall, if the user knowsthe offset between them. The planes and lines are converted to pointpairs, as described below, and the process operates entirely on pointpairs. The points used for registration may be actual scan points or maybe constructed points, such as a corner at the intersection of threeplanes.

Given a set of point pairs, the registration process searches the set ofcandidate points for three pairs that are not colinear. Using the threepoint pairs, one can construct the transformation required to convertthe coordinate system in one view to that used in the other view, whichin turn can be used to transform the scan points to all share a singlecoordinate system. For convenience, the process will be described interms of the first data set remaining fixed while the second data set istransformed to use the coordinate system of the first The process worksequally well if the user fixes the second data set and transforms thefirst.

The first step is to apply a rigid body translation to the second dataset to make the first pair of points coincide in terms of their x, y andz components. The second step is to rotate the second data set about itsfirst point until the lines formed by points one and two in both datasets are colinear. The third step is to rotate the second data set aboutthe line established in the previous step until the planes defined bypoints one, two and three in both data sets are coplanar.

Once an initial estimate is made one can use all the point pairs and anerror minimization method to reduce the sum of the squares of thedistances between each point pair.

In order to use the point registration method described above, the CGP40 uses a novel method to convert lines, planes, and planes with offsetsto sets of point pairs. Whenever a nonzero plane offset is present thenew points introduced are shifted in the second scene to a positionwhere they will match exactly with the corresponding points in the firstscene. The replacement of planes and lines with points makes it simpleto write the error function for minimization, since only point errorsare involved, rather than angular and distance errors simultaneously.

In replacing planes and lines, one can only introduce points that are atlocation relative to the user specified objects, since the origins ofthe two data sets are different. For instance, introducing a new pointpair in a plane at the location closest to the origin would not resultin points that actually match in space, since the origin is arbitrary.However, introducing a point pair at a plane-line intersection will givematching points in the two data sets. Some pairs of objects, likeparallel lines, should not be used to introduce new points so an angulartolerance, called ATOL below, is used to ignore poor object pairs. ATOLis initially set to ten degrees but other values can be used to generatefewer or more artificial point pairs as needed. The point pairs areintroduced in the following order:

For all plane-line pairs where the angle between the line and plane isgreater than ATOL, introduce two new point pairs. The first new point isinserted at the intersection point of the line and plane, and the secondpoint pair is inserted along the line of action at a fixed distance awayfrom the first point, here taken to be the minimum of the line lengthsin the two views.

For all pairs of planes and points, introduce a new point pair on theplane such that the plane normal passes through the new point andspecified point.

For all plane pairs whose normals are at least ATOL apart, generate anew line segment along the intersection of the planes and make the linesegments length equal to the minimum extent that any plane has along theline. The new line segment has no direction, but has both length andposition information. After this step the planes are no longer needed.

For all pairs of lines and points, introduce a new point on the line atthe location where it is closest to the specified point.

For all pairs of lines separated by an angle greater than ATOL,introduce four new pairs of points. The new points are the ends of linesegments along the original line of action, but centered on the locationof closest approach of the two lines. The distance between the new linepoints is equal to the minimum length of the line segment lengths alongthat line of action from the two data sets. After this step the linesare no longer needed.

The result of the plane and line replacements as described above is aset of point pairs that retains the direction information associatedwith the original planes and lines. The augmented set of point pairs canthen be used for the registration process that is described above.

After registration of the two scenes, primitives from the two individualviews which represent the same physical object can be combined using themerging technique described previously. In particular, matching planarpatches representing the same surface can be combined into one extendedplanar patch Similarly, pieces of matching cylindrical surfaces can bemerged to form a single cylinder.

Warping Data Sets

The registration process described above is a rigid body transformationthat does not modify the relative locations of objects within eitherdata set. After registration, most of the point, line or plane pairsthat were identified will still have small errors, since theminimization process reduces the total mean square error. A novel methodis presented that allows the user to force the identified pairs toexactly match by deforming the scene volumes.

As with any measured data, there is some level of error associated witheach scan point location. The magnitude of the error associated with apoint location will vary with the measuring technology used, but someerror will always be present. Since the data under consideration heredescribes surface features of objects, the data errors will manifestthemselves as surface irregularities. For example, a set of pointsacquired from an actual planar surface may not all lie on a plane, butwill have some small scatter away from the real plane location.Calculating a best fit plane through the set of measured points may notgive the real plane location or orientation due to the errors in thepoint data set.

The errors in recovered features, such as planes, cause errors in therelationships between the recovered objects as well. For instance, ifdata is collected from two planes that have an exactly ninety degreeangle between them, the best-fit planes generated from the data pointsmay not be exactly ninety degrees apart. Similarly, cylinders that wereparallel in the real scene may result in best-fit cylinders that are notparallel after fitting from the scanned points. These inconsistencies inthe recovered features, that occur due to measurement errors, willappear whether the data points are collected from a single scan positionor are a union of scans from a variety of different positions.

The lack of fit problem may actually grow if several different sets ofscan data are registered using a relative system. If a series ofsequential scans are collected, and each scan is registered with respectto some recognizable sequence of data points in a previous scan, thenthe absolute errors in each scan may grow. If at the end of the sequenceof scans the locations of features are exactly known, then one mustadjust the scanned data points so that they fit the known locations. Insurveying both the 2-D closure problem and the 3-D benchmark matchingproblems are similar in nature to the problems described above. In thesurveying closure application, when one surveys a sequence of locationsand arrives back at the starting location one typically finds thatthrough cumulative measurement errors the starting and finishinglocations are not at exactly the same location. The closure error, whichis the distance between the starting in finishing locations, isdistributed using well known surveying techniques throughout the otherdata points collected such that the first and last end points meet afterthe correction is made. Similarly, when surveying benchmarks of knownlocation arc introduced into a surveying data set the data set must beadjusted to accommodate the known benchmark locations. Both the closureproblem and the benchmark matching problem can be solved by the methoddescribed here since they can be described in terms of displacementconstraints.

The novel method described here to correct location errors in measured3-D data sets and distributes the errors throughout the point sets byapplying solid mechanics principles to a volume surrounding the datapoints. The method provides a technique for satisfying a wide variety ofdisplacement constraints on 3-D data sets and also distributes themeasurement errors throughout the data sets. The process of deformingthe data sets to achieve these goals is called warping. The displacementconstraints can be specified in terms of both control points, whoseabsolute coordinates are known in space and do not move, and tie points,which represent the same location in two or more data sets, but whoseabsolute location is unknown. One can describe constraints involvingmore complex objects, for instance, line segments by specifying twopoints, and planes by specifying three points. In this way an entirefamily of constraints can be specified and applied to groups of objectssuch as points, lines and planes.

All constraints include an offset term that may be zero or nonzero. Azero offset between points indicates that the points should share thesame final location, but does not prescribe where that location wouldbe. If one of these points were a control point, with a known absoluteposition, then it would not move and the other point in the constraintwould be forced to move to the same location to satisfy the constraintIn all cases, the final location would result from the energyminimization process that is associated with a solid mechanics solution.A non-zero offset between two points indicates that the points are to bea certain distance apart after the warping process is applied.

If the constraint's objects are lines or planes (rather than points),then in addition to the offset one can specify an angle between theobjects. Using this family of constraints types, one could specify anynumber of relations between features in one or more data sets. Theinvention works whether a single or multiple 3-D data sets are involved,and the constraints can be between objects in the same or different datasets.

The solution of the constraint satisfaction problem begins byregistering each of the data sets, as described in the previous section,so that all of the data shares a single coordinate system. Next, solidmechanics theory is applied to a volume surrounding the points in eachdata set to satisfy the displacement constraints. The warping methodoperates on one or more volumes that are created so that they surroundall the points in a given data set. Each of these volumes is consideredto be made out of deformable materials whose properties are specifiable.An isotropic material can be used with an elastic modulus of one and aPoisson's ration of zero. The solid mechanics solution finds the minimumenergy displacement pattern that satisfies the specified constrains.

One can picture each of these volumes made out of a flexible material.If one anchored a volume to prevent rigid body motions and thenprescribed a new location for a point on the interior, one couldenvision a distortion of the volume that involves not only the point ofinterest but also the rest of the volume. In reality, the constraintsthemselves can be used to anchor multiple volumes together. Mechanicsprinciples allow one to determine the minimum energy deformation of thevolume that satisfies the stated constraints, which mimics what wouldactually happen to a real deformable object subjected to the sameconstraints.

In a particular embodiment, the warping method uses principles of solidmechanics to deform a volume containing points of interest in order tosatisfy a set of constraints applied to the data points. Not only arethe constraints satisfied, but the effects of the initial locationerrors are spread throughout the volumes operated on.

The finite element method is used to apply the principles of solidmechanics to the volumes enclosing the points. The volume is discretizedinto a set of points or vertices and a set of elements that connect tothe vertices. Four node tetrahedral elements are used to discretize thevolume.

The first step of the process is to collect the set of constraints thatapply to one or more data sets. During this phase, one must identifyconstraints that are to be satisfied by the warping process. Theseconstraints include the identification of points that represent the samephysical location in different data sets (tie points), like the cornerof a cube, which are to appear at the same location when the warpingprocess is completed. Some of the tie points may not be points that werescanned in the original data set, but may be constructed from groups ofother points. For instance, if one had a series of points thatrepresented three planes intersecting at a corner, then one could fitthree planes to the points, and use the resulting corner point as a tiepoint. The constraints are specified in terms of pairs of objects, suchas points, lines and planes, as well as the desired offset and anglebetween them. The two objects involved in the constraint can becontained in a single data set or can occur in different data sets.Within a single data set, one could specify that lines or planes remainparallel, or that the distance between two points be a specified amount.Between multiple data sets, one could write similar constraints, orindicate that the features seen in two data sets represent the sameobject. One might also know the actual location of some points veryaccurately (benchmarks) and constrain points in the data set to lie atthe known locations. Using these benchmark points to anchor differentpoints in the data sets enables the closure problem to be solved, sincethe data sets will be warped so that the measured data points moveexactly to the desired control point locations and the errors in thedata set will be smoothed over all volumes.

The second step in the warping process is to register all the data setsinvolved, as described in the previous section.

The third step in the warping process is to select a volume thatsurrounds the region of interest, and describe the volume in terms of aset of new points. The region that can be scanned by the FDV 10 iscalled the view volume and is shaped like a pyramid, with the tip of thepyramid located at the origin of the scanning device. A pyramidal shapecan be used to bound the view region for the purpose of warping, and thepyramid is easily described by five points, using the same coordinatesystem as the data points. These new points do not become part of thedata set, but are used in the warping process. The convex hull of thesepoints represents the new volume surface, and should enclose all thedata points on the interior. This operation is performed separately foreach data set.

The fourth step is to mesh each of the data volumes. Meshing involvesfilling the volume with finite elements that leave no gaps and that donot overlap. The finite elements span between the points or verticesthat have been defined on the volume boundary and those that areinvolved in constraints on the interior. The points in the data set donot all need to be included in the warping process, only those that areused in constraint specifications and those that define the volumeboundary need to be used. The elements in the initial mesh may be ofpoor quality due to their shape. Long sliver elements, for instance, areknown to give poor results in finite element analysis. Therefore, themeshing process is actually iterative. New points are inserted into themesh, and then old elements are removed and new elements are introducedso that the mesh quality improves. This iterative process continuesuntil one is satisfied with the overall quality of the mesh. In onepreferred embodiment, four node tetrahedral elements are used. Theinitial mesh is constructed by applying a 3-D Delaunay triangulation onthe starting set of points. The iterative process identifies poorlyshaped elements using an element quality measure, and introduces newpoints and remeshes the region. The process terminates when all elementsmeet a minimum quality criteria. The preferred implementation useslongest edge bisection to introduce new points that improve the mesh,but other methods can be used.

The fifth step processes the constraints described in step one into asystem of linear constraints. In the preferred embodiment, the finalsystem of constraints is linear in terms of the nodal displacements atthe vertices of the tetrahedral elements. The desired form of theconstraints is:

Cu=q  (6)

The matrix C contains constant coefficients. The number of rows of C isequal to the number of constraints in the system. The vector urepresents the 3-D displacements of the vertices of the tetrahedralelements. The vector q contains constant coefficients. If theconstraints are homogenous then each element of q will be 0. The form ofconstraint specification given in Equation (6) allows arbitrary linearmultipoint (involving more than one vertex) constraints.

The conversion of the constraints specified in step one into the formshown above depends on the type of constraints involved. For two pointsto be tied together the constraint would be:

p ₁ +u ₁ =p ₂ +u ₂  (7)

or

u ₁ −u ₂ −p ₂ −p ₁  (8)

In these equations, p₁ and p₂ are vectors from the origin to thevertices of interest, while u₁ and u₂ are the displacements of the samevertices during warping. Equation (7) demands that the final location ofeach vertex, equal to the starting point plus the displacement duringwarping, be equal to the final location of the other point. Equation (8)is in the form of Equation (6), with q=p₂−p₁ and results in three linearconstraints, in terms of the x, y, and z components of the nodaldisplacements. Expanding Equation (8) into three equations in the formof Equation (6) then gives: $\begin{matrix}{{\lbrack \quad \begin{matrix}1 & 0 & 0 & {- 1} & 0 & 0 \\0 & 1 & 0 & 0 & {- 1} & 0 \\0 & 0 & 1 & 0 & 0 & {- 1}\end{matrix}\quad \rbrack \quad \begin{Bmatrix}u_{1x} \\u_{1y} \\u_{1z} \\u_{2x} \\u_{2y} \\u_{2z}\end{Bmatrix}} = \begin{Bmatrix}0 \\0 \\0\end{Bmatrix}} & (9)\end{matrix}$

Other constraints, like the distance between two points, are non-linearin nature. The nonlinear constraints can use the existing geometry ofthe system as well as small deformation assumptions to produce linearmultipoint constraints. For example, to specify the desired distancebetween two points to be some specified value x, one could determine thevector v₂₁ between the final points locations:

v ₂₁=(p ₂ +u ₂)−(p ₁ +u ₁)  (10)

and then specify the desired length of the vector:

∥v₂₁∥=x  (11)

or, using the vector dot product:

v ₂₁ ·v ₂₁ =x ²  (12)

Both Equations (11) and (12) are nonlinear in terms of the displacementsof the nodes, u₁ and u₂. To linearize the constraint we can specify thedesired length along the original line of action to be equal to thedesired offset: $\begin{matrix}{{{let}\quad n_{21}} = \frac{( {p_{2} - p_{1}} )}{{p_{2} - p_{1}}}} & (13)\end{matrix}$

v₂₁·n₂₁=x  (14)

or

[(p ₂ +u ₂)−(p ₁ +u ₁)]·n ₂₁ =x  (15)

u ₂ ·n ₂₁ −u ₁ ·n ₂₁ =x−∥p ₂ −p ₁∥  (16)

The term on the right hand side of equation (16) is the desired distancebetween the points minus the current distance between the points. The x,y and z components of n₂₁ are constraints. Equation (16) can be writtenas a single constraint in the proper form as: $\begin{matrix}{{{\langle\quad \begin{matrix}{- n_{21x}} & {- n_{21y}} & {- n_{21z}} & n_{21x} & n_{21y} & n_{21z}\end{matrix}\quad\rangle}\begin{Bmatrix}u_{1x} \\u_{1y} \\u_{1z} \\u_{2x} \\u_{2y} \\u_{2z}\end{Bmatrix}} = {x - {{p_{2} - p_{1}}}}} & (17)\end{matrix}$

In Step 6 the final system of linear equations is assembled. There aretwo parts to this step: first, assembling the element stiffnesses foreach of the tetrahedral elements, and second, selecting and applying aconstraint handling technique. The calculation and assembly of elementstiffnesses follow standard finite element procedures. Using constraintsin the form of Equation (6) involves a constraint processing method. TheLagrange Multipliers technique can be used to introduce the effect ofthe linear constraints, but any other method, such as penalty ortransformation techniques, could be used equally effectively.

Using Lagrange Multipliers, one introduces a new variable into the finalsystem of equations for each constraint in the system. One then modifiesthe static equilibrium equations for the unconstrained system, which aregiven by:

Ku=r  (18)

In Equation (18), K is the system stiffness matrix, assembled from theindividual element stiffness contributions, u is the displacement vectorthat is the solution of the problem, and r is a vector of externallyapplied loads. In this embodiment of the invention, there are noexternally applied loads, so the r vector contains only zeroes. Equation(18) does not include the effect of any constraints, but these can beincluded using the Lagrange Multipliers technique to give the system ofequations: $\begin{matrix}{{\begin{bmatrix}K & C^{T} \\C & 0\end{bmatrix}\begin{Bmatrix}u \\u_{L}\end{Bmatrix}} = \begin{Bmatrix}r \\q\end{Bmatrix}} & (19)\end{matrix}$

In Equation (19) K, C, u, r, and q are as previously defined, and u_(L)is a vector containing the additional Lagrange Multiplier variables thatare introduced using this method. The matrix C^(T) is the transpose ofC, and 0 is a matrix of zeroes. The solution of Equation (19) gives thedisplacements u that satisfy the linear constraints described by C andq. Note that these constraints may be linearizations of nonlinearconstraints, and the nonlinear constraints might not be satisfied by thesolution at this point.

If penalty or transformation methods were used instead of LagrangeMultipliers, a system of linear equations different from those shown inEquation (19) would be produced, but the solution of the linear systemof equations will give similar values for the displacement vector u.

In Step 7 Equation (19) is solved to give u and u_(L). There are manymethods available to solve large systems of linear equations, and thepreferred embodiment uses a symmetric solver with a profile storagescheme. The different types of solvers that could be used giveessentially the same results, but optimize speed and memory usagedifferently.

The preferred embodiment uses a direct solver, but iterative sparsesolvers could be used as well. The system of equations shown in Equation(19) is sparse, so significant speed enhancements can be achieved byselecting the proper solver. However, the results of the warping processoverall are unaffected by this choice.

In Step 8, one must check if the current displacement satisfies theconstraints to a desired level of accuracy. If the current deformedshape violates the offset or angle in any of the constraints collectedin Step 1 by more than a user specified tolerance, then steps 5 through7 must be repeated, starting with the new deformed shape. Thelinearizations of the shape may change on each iteration since thegeometry of the volume changes with the cumulative deformations. Whenall constraints are satisfied within the given tolerance, then one canproceed to step 9.

Step 9 uses the nodal deformations u calculated in Step 7 to determinethe deformation of any point of interest within the volumes. For eachpoint of interest, one must find an finite element that includes thepoint on its surface or interior. If the point is internal to an elementthen only one such element exists. If the point is on the surface of anelement or along the edge of an element, then several elements could beconsidered to contain the point. Any of these elements can be selectedto determine where the point of interest moves. If the point is sharedbetween elements, then the use of any of the elements to find the pointdisplacement will give the same results. Once an element is identified,the vertex displacements of that element are extracted from u and areused to determine the displacement of any point on the interior using aninterpolation process. This procedure uses the finite element shapefunctions which are linear in the preferred embodiment, and is a commonoperation in finite element analysis.

Auto-Segmentation

The novel auto-segmentation process, as presented below, involves asimilar sequence of operations to the manual modeling process describedpreviously. A point cloud is segmented, geometric primitive objects arefit to the point groups, and then modeling operations, such as extensionand intersection are used to complete the model. In this novel process,automation is applied to each of these steps, as well as the entireprocess, as described below.

It is possible to automatically partition the scan points into groupsrepresenting primitive geometrical shapes by using variations of commonmachine vision techniques. A gridded scan field is stored in a twodimensional array of points, much like a regular bitmap. The scan fielddiffers from a bitmap in that more information is stored at eachlocation than just a color. Each point stores its location in space,from which the distance to the scanner can be calculated, as well as theintensity of the return laser pulse. The depth information calculatedfrom the three dimensional position stored at the points is crucial tothe automated segmentation algorithm described here, even though manyoperations, such as filtering, rating, thresholding and thinning arecommonly used image manipulation operations.

The first stage of the auto-segmentation process is to estimate thesurface normal at each point in the grid This can be achieved using manydifferent techniques, the current embodiment of the software fits aplane to the nearest neighbors of the point in the 3 by 3 gridsurrounding it The normal of the resulting plane is taken as the normalat the center point. Each point in the grid has a normal calculated inthe same way, except that edge and corner points ignore the missingneighbors in the normal calculation. The normal stored at each point isa three dimensional vector and is normalized to have unit length.

In the second phase two rating images are created by convolving standardedge detection filters over the grid. The first rating image is createdby convolving the depth of the grid point with an edge detection filterto identify depth discontinuities, such as those that would occur at anoccluded edge. A variety of edge detection filters can be used, butrather than operate on color or intensity the filter operates on thedepth information stored at each grid point.

The second rating image is created by convolving the normal with an edgedetection filter. The normal rating image is actually composed of 3subimages created from a convolution with the normal's x, y, and zcomponents. The resulting three values are combined by taking the squareroot of the sum of the squares to give a per-point scalar value. Thesecond rating image is used to identify normal discontinuities, such asthose that would occur at the edge between a wall and a floor. Again, awide variety of edge detection filters can be used, but the values usedare normal coefficients rather than color or intensity.

Once the two rating images have been created they must separately beconverted to binary images. Conventional machine vision algorithms, suchas recursive thresholding can be used to achieve this task. Each pointin the depth and normal rating images contains an estimate of thegradient of the depth and normal respectively. Recursive thresholdingcan be used to isolate the regions of highest gradient In the resultingbinary images the points in the regions of highest gradient are markedas edge points while the rest of the points are marked as non-edge.

A final binary image is created by marking a point as an edge point ifit is marked as an edge point in either or both of the two binary imagescreated by recursive thresholding above. All other points are marked asnon-edge. This image contains all edge points that delineate theboundaries between groups of points on different surfaces.

The final step of the point partitioning process is to use a connectedcomponents algorithm to collect the points into groups separated byedges. Points are considered to be connected only if they are verticallyor horizontally adjacent in the grid, diagonal adjacency is not used.Very simple algorithms can be used to identify the unique groups ofnon-edge points in the image. Each group of connected points is then cutfrom the initial point set to form a new group of points. The result ofthis algorithm is the partitioning of the point set into multiple pointgroups that each represents a single surface. Each of the new pointgroups can be fit by a geometric primitive as described in the nextsection.

Once the scan cloud has been partitioned into groups of scan points thatlie on different surfaces, the next step is to fit objects to thedesired surfaces. A variety of methods can be used to achieve this task.The current embodiment of the software can perform the object fittingprocess in two different ways. The first method fits a series of objectsto each group of points, and selects the objects that produces thesmallest distance errors between the measured points and the fittedobject surfaces. The second method uses the quadric surface fitdescribed previously, and resulting principle curvatures, to determineif a plane, cylinder or sphere should be fit to a particular pointgroup. Other variations of these approaches could also be used, such asprogressive commitment, where objects are fitted in order from simplestto most complicated, and the process stops whenever the errorsassociated with the particular fit drop to acceptable levels.

The last stage of auto-segmentation process extends primitive objects,where possible, to create complete object intersections, rather thanstopping at scan point boundaries. Using the gridded nature of theoriginal data and the edge information from the point partitioningalgorithm described above, it is possible to extend and intersectobjects. For all edges that result from surface intersections, which arethe surface normal discontinuity edges described above, one can extendthe objects on either side of the edge to form an intersection.

Model Annotation

In order to compose a semantically rich 3-D model, individual parts inthe above geometrical model can be annotated with additional, possiblynon-geometric, information, such as material references or part numbers.Tthis information can be entered manually through a special window fordisplaying object attributes.

The user may click on an individual part in the geometrical model andrecover such additional information through other windows. Similarly,the user may request that all parts which meet some selection criteriaare to be highlighted.

A novel method is also used for automatic model annotation. This methoduses the FDV 10 to scan bar codes containing pertinent informationrelating to any given object. Standard bar code reading and decodingtechniques are used to convert optical information to useful digitalinformation that is associated with a given object scanned at the sametime as the bar code. The captured information can be displayed asdescribed above for the manual method.

Geometry Display and Query

The model is accessible in a variety of ways, including access throughthe data window 1610 where the model is rendered. Many standard graphicinterface techniques can be used to manipulate the view of the model; inone preferred embodiment, a crystal ball interface is used. Any objectin view can be selected using the pointing device; its geometricalattributes can then be displayed, in addition to other annotations addedto the object. Traversing the data set can be simplified by placingobjects in different layers, and then displaying only the layers ofinterest. Reducing the number of rendered objects in this way increasesthe interactive performance of the program. In addition to querying thegeometric properties of individual objects several standard tools formeasuring distances and angles between objects can be employed.Additional standard techniques can be used for operations such ascontour generation, 2-D section cutting, and automated dimensioning.

The resulting model can be exported to any of a number of CAD programsfor further editing or designing. In the preferred embodiment, the CGP40 can create a CAD file in a format compatible with several commercialCAD programs, and can then start the CAD program, having it load thegenerated data file. With both the CGP and the CAD program running, theuser can then view and work with the model using either program.

The following documents form an integral part of this specification:

Modular Decomposition

Summary of Proposed CGP Specification

Cyrax Software Specification

Product Overview

Collection of View Slides

Introduction and Overview

The remainder the specification as originally filed appears in PCT WO97/40342, published on Oct. 30, 1997, at pages 69-122 and isincorporated herein by reference.

What is claimed is:
 1. A method for fitting a point cloud representing acorner, comprising: determining a fit of three planes to the points ofthe point cloud and creating the planes for a model; determining thethree lines at the intersection of pairs of planes and creating thelines for the model; determining the vertex point at the intersection ofthe three planes and creating a vertex point for the model; and whereinthe determining said fit of three planes, includes: using a firstfitting algorithm to determine a first plane of the corner whichcorresponds to a first set of points of the group of points; using asecond fitting algorithm to determine a second plane of the corner whichcorresponds to a second set of points of the group of points; and usinga third fitting algorithm to determine a third plane of the corner whichcorresponds to a third set of points of the group of points.
 2. A methodfor fitting a point cloud representing a corner, comprising:partitioning a group of points specified by a user into a first set ofpoints, wherein the first set of points corresponds to a first planethat forms a first part of the corner; partitioning the group of pointsspecified by said user into a second set of points, wherein the secondset of points corresponds to a second plane that forms second part ofthe corner; and partitioning the group of points specified by said userinto a third set of points wherein the third set of point correspond toa third plane that forms a third part of the corner.
 3. The method ofclaim 2, further comprising: using a first fitting algorithm todetermine the first plane of the corner which corresponds to the firstset of points of the group of points; using a second fitting algorithmto determine the second plane of the corner which corresponds to thesecond set of points of the group of points; and using a third fittingalgorithm to determine the third plane of the corner which correspondsto the third set of points of the group of points.